[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:04 dist-test-slave-jkn7 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:04 dist-test-slave-jkn7 krb5kdc[395](info): set up 2 sockets
Jun 21 02:38:04 dist-test-slave-jkn7 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 02:38:05.854013 388 mini_kdc.cc:121] Time spent starting KDC: real 2.423s user 0.008s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:05 dist-test-slave-jkn7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:06.018011 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38973
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:06.341817 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:06.342125 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:06.342216 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:06.349861 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:06.349983 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:06.350028 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:06.350078 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:06.350122 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:06.360073 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:06.361439 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:06.362709 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:06.374868 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:06.374891 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:06.374897 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:06.375792 408 server_base.cc:1034] running on GCE node
I20260621 02:38:06.376616 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:06.378278 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:06.379505 408 hybrid_clock.cc:648] HybridClock initialized: now 1782009486379439 us; error 85 us; skew 500 ppm
I20260621 02:38:06.379866 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:06 dist-test-slave-jkn7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:06.383126 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:06.384572 408 webserver.cc:460] Webserver started at http://127.0.97.62:36309/ using document root <none> and password file <none>
I20260621 02:38:06.385218 408 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:06.385377 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:06.385653 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:06.389137 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "9e703daacc9c40cfaee50594e26430c7"
format_stamp: "Formatted at 2026-06-21 02:38:06 on dist-test-slave-jkn7"
server_key: "3d163459ffc3aad5644ba450af4c6f73"
server_key_iv: "cf5280270b67f5ecf8d92d944cd24fd7"
server_key_version: "encryptionkey@0"
I20260621 02:38:06.389942 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "9e703daacc9c40cfaee50594e26430c7"
format_stamp: "Formatted at 2026-06-21 02:38:06 on dist-test-slave-jkn7"
server_key: "3d163459ffc3aad5644ba450af4c6f73"
server_key_iv: "cf5280270b67f5ecf8d92d944cd24fd7"
server_key_version: "encryptionkey@0"
I20260621 02:38:06.395156 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:38:06.398468 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:06.399832 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:38:06.399976 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "9e703daacc9c40cfaee50594e26430c7"
format_stamp: "Formatted at 2026-06-21 02:38:06 on dist-test-slave-jkn7"
server_key: "3d163459ffc3aad5644ba450af4c6f73"
server_key_iv: "cf5280270b67f5ecf8d92d944cd24fd7"
server_key_version: "encryptionkey@0"
I20260621 02:38:06.400141 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:06.439134 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:06.443647 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:06.443883 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:06.462563 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38973
I20260621 02:38:06.462595 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38973 every 8 connection(s)
I20260621 02:38:06.463936 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:06.468029 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:06.474712 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 408
I20260621 02:38:06.475054 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:06.476877 388 external_mini_cluster.cc:1428] Setting key 173c1e73d5e980ff4e618e7a85664559
I20260621 02:38:06.478150 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7: Bootstrap starting.
I20260621 02:38:06.482075 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:06.483603 475 log.cc:824] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:06.486196 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7: No bootstrap required, opened a new log
I20260621 02:38:06.494654 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e703daacc9c40cfaee50594e26430c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260621 02:38:06.495086 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:06.495226 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e703daacc9c40cfaee50594e26430c7, State: Initialized, Role: FOLLOWER
I20260621 02:38:06.495934 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e703daacc9c40cfaee50594e26430c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260621 02:38:06.496173 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:06.496321 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:06.496512 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:06.498847 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e703daacc9c40cfaee50594e26430c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260621 02:38:06.499356 475 leader_election.cc:302] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e703daacc9c40cfaee50594e26430c7; no voters:
I20260621 02:38:06.500079 475 leader_election.cc:288] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:06.500226 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:06.501892 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [term 1 LEADER]: Becoming Leader. State: Replica: 9e703daacc9c40cfaee50594e26430c7, State: Running, Role: LEADER
I20260621 02:38:06.502450 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e703daacc9c40cfaee50594e26430c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } }
I20260621 02:38:06.502755 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:06.505504 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e703daacc9c40cfaee50594e26430c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e703daacc9c40cfaee50594e26430c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } } }
I20260621 02:38:06.505895 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:06.511148 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 21 02:38:06 dist-test-slave-jkn7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:06.511407 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e703daacc9c40cfaee50594e26430c7. Latest consensus state: current_term: 1 leader_uuid: "9e703daacc9c40cfaee50594e26430c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e703daacc9c40cfaee50594e26430c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38973 } } }
I20260621 02:38:06.512974 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:06.517674 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:06.529094 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:06.480698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.0.97.62:38973)
0621 02:38:06.481518 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:06.481529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:06.483031 (+ 1502us) server_negotiation.cc:407] Connection header received
0621 02:38:06.494137 (+ 11106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:06.494162 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:06.494576 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:06.495008 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:06.498239 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:06.499692 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:06.502835 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:06.504063 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:06.504314 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:06.513816 (+ 9502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:06.513830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:06.513853 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:06.513911 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:06.519522 (+ 5611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:06.521003 (+ 1481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:06.521016 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:06.521031 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:06.521340 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:06.522035 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:06.522044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:06.522053 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:06.522890 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:06.523332 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:06.526625 (+ 3293us) server_negotiation.cc:299] Negotiation successful
0621 02:38:06.527188 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":386,"threads_started":1}
I20260621 02:38:06.538264 485 catalog_manager.cc:1347] Generated new cluster ID: 15596b123dd94af28e7729ba3a4bed6f
I20260621 02:38:06.538400 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:06.550298 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:06.551723 485 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:06.557693 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e703daacc9c40cfaee50594e26430c7: Generated new TSK 0
I20260621 02:38:06.558514 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:38:06.609092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:06.963846 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:06.964119 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:06.964214 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:06.971361 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:06.971455 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:06.971562 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:06.981269 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:06.982492 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:06.983703 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:06.997084 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:06.997108 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:06.997449 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:06.997905 502 server_base.cc:1034] running on GCE node
I20260621 02:38:06.998499 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:06.999377 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:07.000622 502 hybrid_clock.cc:648] HybridClock initialized: now 1782009487000561 us; error 54 us; skew 500 ppm
I20260621 02:38:07.000980 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:06 dist-test-slave-jkn7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:07.004462 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:38:07.005940 502 webserver.cc:460] Webserver started at http://127.0.97.1:40463/ using document root <none> and password file <none>
I20260621 02:38:07.006647 502 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:07.006783 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:07.007086 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:07.010303 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "7d7e9ba616ab4fbfb6ff95b566b294fc"
format_stamp: "Formatted at 2026-06-21 02:38:07 on dist-test-slave-jkn7"
server_key: "ed24678ef3685b3ab4579c5c5925eda8"
server_key_iv: "d80c4f72a6939e943c959b9cf1fdad85"
server_key_version: "encryptionkey@0"
I20260621 02:38:07.011075 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7d7e9ba616ab4fbfb6ff95b566b294fc"
format_stamp: "Formatted at 2026-06-21 02:38:07 on dist-test-slave-jkn7"
server_key: "ed24678ef3685b3ab4579c5c5925eda8"
server_key_iv: "d80c4f72a6939e943c959b9cf1fdad85"
server_key_version: "encryptionkey@0"
I20260621 02:38:07.016083 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:38:07.019394 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:07.020573 502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 02:38:07.020735 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "7d7e9ba616ab4fbfb6ff95b566b294fc"
format_stamp: "Formatted at 2026-06-21 02:38:07 on dist-test-slave-jkn7"
server_key: "ed24678ef3685b3ab4579c5c5925eda8"
server_key_iv: "d80c4f72a6939e943c959b9cf1fdad85"
server_key_version: "encryptionkey@0"
I20260621 02:38:07.020941 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:07.031071 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:07.035064 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:07.035332 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:07.036170 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:07.037496 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:07.037593 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:07.037729 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:07.037798 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:07.072403 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33833
I20260621 02:38:07.072443 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33833 every 8 connection(s)
I20260621 02:38:07.073738 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:07.077621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 502
I20260621 02:38:07.077840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:07.078475 388 external_mini_cluster.cc:1428] Setting key c70e4da4d94271109e7db676730fc782
Jun 21 02:38:07 dist-test-slave-jkn7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260621 02:38:07.095894 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:07.076149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60863 (local address 127.0.97.62:38973)
0621 02:38:07.076593 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:07.076599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:07.077646 (+ 1047us) server_negotiation.cc:407] Connection header received
0621 02:38:07.079762 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:07.079766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:07.079844 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:07.079942 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:07.081603 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:07.083173 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:07.084696 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:07.085430 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:07.085581 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:07.088947 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:07.088956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:07.088964 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:07.089005 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:07.092988 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:07.093725 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:07.093732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:07.093735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:07.093874 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:07.094277 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:07.094284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:07.094290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:07.094665 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:07.094826 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:07.095490 (+ 664us) server_negotiation.cc:299] Negotiation successful
0621 02:38:07.095640 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":213,"threads_started":1}
I20260621 02:38:07.096797 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:07.076763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.1:60863)
0621 02:38:07.077498 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:07.077537 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:07.079382 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:07.080241 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:07.080254 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:07.080840 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:07.081443 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:07.081461 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:07.083325 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:07.083329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:07.084561 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:07.084572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:07.085627 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:07.085632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:07.085817 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:07.086495 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:07.086516 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:07.088743 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:07.093130 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:07.093136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:07.093147 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:07.093609 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:07.093990 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:07.093993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:07.093995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:07.094152 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:07.094787 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:07.094792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:07.095076 (+ 284us) client_negotiation.cc:769] Sending connection context
0621 02:38:07.095354 (+ 278us) client_negotiation.cc:240] Negotiation successful
0621 02:38:07.095703 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":288,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260621 02:38:07.098435 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38973
I20260621 02:38:07.098745 630 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:07.099411 630 heartbeater.cc:511] Master 127.0.97.62:38973 requested a full tablet report, sending...
I20260621 02:38:07.101450 440 ts_manager.cc:194] Registered new tserver with Master: 7d7e9ba616ab4fbfb6ff95b566b294fc (127.0.97.1:33833)
I20260621 02:38:07.103698 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:38:07.120283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:07.441197 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:07.441475 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:07.441568 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:07.448376 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:07.448479 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:07.448593 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:07.458165 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:07.459389 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:07.460601 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:07.474133 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:07.474120 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:07.474380 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:07.475659 636 server_base.cc:1034] running on GCE node
I20260621 02:38:07.476191 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:07.477451 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:07.478703 636 hybrid_clock.cc:648] HybridClock initialized: now 1782009487478652 us; error 37 us; skew 500 ppm
I20260621 02:38:07.479126 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:07 dist-test-slave-jkn7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:07.483408 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:38:07.487133 636 webserver.cc:460] Webserver started at http://127.0.97.2:45921/ using document root <none> and password file <none>
I20260621 02:38:07.488188 636 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:07.488348 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:07.488771 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:07.491598 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "aacf02d2dbe24fb6b01d0f83302e8977"
format_stamp: "Formatted at 2026-06-21 02:38:07 on dist-test-slave-jkn7"
server_key: "a73f955c77a6d9402a9a511cd9a588ff"
server_key_iv: "56d5cc095d24e0780dd5d246d4dae8eb"
server_key_version: "encryptionkey@0"
I20260621 02:38:07.492375 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "aacf02d2dbe24fb6b01d0f83302e8977"
format_stamp: "Formatted at 2026-06-21 02:38:07 on dist-test-slave-jkn7"
server_key: "a73f955c77a6d9402a9a511cd9a588ff"
server_key_iv: "56d5cc095d24e0780dd5d246d4dae8eb"
server_key_version: "encryptionkey@0"
I20260621 02:38:07.497414 636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:38:07.500643 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:07.501987 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:38:07.502159 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "aacf02d2dbe24fb6b01d0f83302e8977"
format_stamp: "Formatted at 2026-06-21 02:38:07 on dist-test-slave-jkn7"
server_key: "a73f955c77a6d9402a9a511cd9a588ff"
server_key_iv: "56d5cc095d24e0780dd5d246d4dae8eb"
server_key_version: "encryptionkey@0"
I20260621 02:38:07.502363 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:07.514746 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:07.518817 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:07.519096 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:07.519963 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:07.521556 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:07.521670 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:07.521795 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:07.521857 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:07.557204 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44075
I20260621 02:38:07.557256 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44075 every 8 connection(s)
I20260621 02:38:07.558490 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:07.565095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 636
I20260621 02:38:07.565338 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:07.565760 388 external_mini_cluster.cc:1428] Setting key 8d15bf765d8cf36a00b07b36f38fa2d5
Jun 21 02:38:07 dist-test-slave-jkn7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:07.579794 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:07.560984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37291 (local address 127.0.97.62:38973)
0621 02:38:07.561147 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:07.561152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:07.562280 (+ 1128us) server_negotiation.cc:407] Connection header received
0621 02:38:07.564427 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:07.564432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:07.564528 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:07.564634 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:07.566231 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:07.567048 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:07.568689 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:07.569106 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:07.569235 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:07.572609 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:07.572617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:07.572626 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:07.572669 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:07.576244 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:07.576975 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:07.576983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:07.576990 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:07.577137 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:07.577748 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:07.577756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:07.577763 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:07.578163 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:07.578320 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:07.579082 (+ 762us) server_negotiation.cc:299] Negotiation successful
0621 02:38:07.579343 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":58}
I20260621 02:38:07.580499 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:07.561478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.2:37291)
0621 02:38:07.562131 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:07.562166 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:07.564156 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:07.564860 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:07.564868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:07.565485 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:07.566075 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:07.566093 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:07.567197 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:07.567207 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:07.568544 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:07.568555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:07.569355 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:07.569363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:07.569573 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:07.570235 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:07.570257 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:07.572392 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:07.576406 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:07.576413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:07.576426 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:07.576831 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:07.577280 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:07.577370 (+ 90us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:07.577373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:07.577574 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:07.578304 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:07.578320 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:07.578670 (+ 350us) client_negotiation.cc:769] Sending connection context
0621 02:38:07.578917 (+ 247us) client_negotiation.cc:240] Negotiation successful
0621 02:38:07.579326 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":260,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:38:07.582055 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38973
I20260621 02:38:07.582401 764 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:07.583119 764 heartbeater.cc:511] Master 127.0.97.62:38973 requested a full tablet report, sending...
I20260621 02:38:07.584570 440 ts_manager.cc:194] Registered new tserver with Master: aacf02d2dbe24fb6b01d0f83302e8977 (127.0.97.2:44075)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260621 02:38:07.585460 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:38:07.605949 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:07.924659 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:07.924942 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:07.925165 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:07.932365 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:07.932467 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:07.932641 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:07.942335 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38973
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:07.943727 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:07.945125 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:07.958606 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:07.958688 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:07.958644 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:07.962546 769 server_base.cc:1034] running on GCE node
I20260621 02:38:07.963106 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:07.963956 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:07.965190 769 hybrid_clock.cc:648] HybridClock initialized: now 1782009487965144 us; error 31 us; skew 500 ppm
I20260621 02:38:07.965540 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:07 dist-test-slave-jkn7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:07.968497 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:38:07.970479 769 webserver.cc:460] Webserver started at http://127.0.97.3:46623/ using document root <none> and password file <none>
I20260621 02:38:07.971171 769 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:07.971264 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:07.971609 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:07.974455 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "da5f473ede42448f893a4fe230023f68"
format_stamp: "Formatted at 2026-06-21 02:38:07 on dist-test-slave-jkn7"
server_key: "e1015d1d1fa4a18baa3ad699578652a3"
server_key_iv: "e9c833ec97dca13019de6e10df69bb6d"
server_key_version: "encryptionkey@0"
I20260621 02:38:07.975224 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "da5f473ede42448f893a4fe230023f68"
format_stamp: "Formatted at 2026-06-21 02:38:07 on dist-test-slave-jkn7"
server_key: "e1015d1d1fa4a18baa3ad699578652a3"
server_key_iv: "e9c833ec97dca13019de6e10df69bb6d"
server_key_version: "encryptionkey@0"
I20260621 02:38:07.980085 769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:38:07.983510 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:07.984769 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260621 02:38:07.984921 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "da5f473ede42448f893a4fe230023f68"
format_stamp: "Formatted at 2026-06-21 02:38:07 on dist-test-slave-jkn7"
server_key: "e1015d1d1fa4a18baa3ad699578652a3"
server_key_iv: "e9c833ec97dca13019de6e10df69bb6d"
server_key_version: "encryptionkey@0"
I20260621 02:38:07.985121 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:08.024945 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:08.029245 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:08.029618 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:08.030622 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:08.031993 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:08.032070 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:08.032191 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:08.032248 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:08.068379 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34101
I20260621 02:38:08.068467 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34101 every 8 connection(s)
I20260621 02:38:08.069676 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:08.072101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 769
I20260621 02:38:08.072306 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:08.073308 388 external_mini_cluster.cc:1428] Setting key cb2b7737358e8ba18010fcb37dac7889
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:08.091130 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:08.073376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58769 (local address 127.0.97.62:38973)
0621 02:38:08.073572 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:08.073580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:08.075110 (+ 1530us) server_negotiation.cc:407] Connection header received
0621 02:38:08.077177 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:08.077182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:08.077263 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:08.077410 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:08.078837 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.079571 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.081096 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.081534 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.081629 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:08.084993 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:08.084998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:08.085001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:08.085034 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:08.087609 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.088509 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.088516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.088520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.088666 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.089164 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.089170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.089173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.089623 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:08.089775 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:08.090543 (+ 768us) server_negotiation.cc:299] Negotiation successful
0621 02:38:08.090773 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:38:08.092131 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:08.074099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38973 (local address 127.0.97.3:58769)
0621 02:38:08.074952 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:08.075002 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:08.076925 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:08.077663 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:08.077671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:08.078157 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:08.078679 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:08.078695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.079725 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.079729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:08.080976 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:08.080987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.081767 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.081777 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:08.082033 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:08.082641 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:08.082662 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:08.084770 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:08.087772 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:08.087781 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:08.087797 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:08.088368 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:08.088840 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:08.088844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:08.088848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:08.089048 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:08.089763 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:08.089771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:08.090151 (+ 380us) client_negotiation.cc:769] Sending connection context
0621 02:38:08.090448 (+ 297us) client_negotiation.cc:240] Negotiation successful
0621 02:38:08.090982 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":366,"threads_started":1}
I20260621 02:38:08.094018 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38973
I20260621 02:38:08.094323 897 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:08.095039 897 heartbeater.cc:511] Master 127.0.97.62:38973 requested a full tablet report, sending...
I20260621 02:38:08.096524 440 ts_manager.cc:194] Registered new tserver with Master: da5f473ede42448f893a4fe230023f68 (127.0.97.3:34101)
I20260621 02:38:08.097436 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58769
I20260621 02:38:08.102257 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:38:08.107319 630 heartbeater.cc:503] Master 127.0.97.62:38973 was elected leader, sending a full tablet report...
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009488, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009488, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:08.146448 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:08.130681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.0.97.1:33833)
0621 02:38:08.131421 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:08.131429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:08.131458 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:38:08.131519 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:08.131525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:08.131741 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:08.131895 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:08.133225 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.134306 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.135577 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.136184 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.136292 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:08.140741 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:08.140755 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:08.140771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:08.140832 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:08.143557 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.144307 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.144314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.144317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.144446 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.144819 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.144825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.144828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.145185 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:08.145522 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:08.145811 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 02:38:08.146081 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":448,"threads_started":1}
W20260621 02:38:08.147763 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37778
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:08.190172 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:08.176704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.0.97.1:33833)
0621 02:38:08.176960 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:08.176968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:08.177159 (+ 191us) server_negotiation.cc:407] Connection header received
0621 02:38:08.177347 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:08.177356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:08.177426 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:08.177517 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:08.178751 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.179441 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.180587 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.181017 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.181116 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:08.185272 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:08.185277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:08.185321 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:08.185395 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:08.187704 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.188312 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.188316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.188318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.188433 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.188816 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.188823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.188826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.189065 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:08.189352 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:08.189657 (+ 305us) server_negotiation.cc:299] Negotiation successful
0621 02:38:08.189866 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260621 02:38:08.191653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 502
I20260621 02:38:08.204670 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:08.192581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.0.97.1:33833)
0621 02:38:08.192774 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:08.192782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:08.193043 (+ 261us) server_negotiation.cc:407] Connection header received
0621 02:38:08.193229 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:08.193233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:08.193336 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:08.193435 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:08.194566 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.195248 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.196491 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.196927 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.197068 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:08.199484 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:08.199489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:08.199493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:08.199523 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:08.202014 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.202688 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.202700 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.202703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.202830 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.203216 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.203224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.203228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.203527 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:08.203770 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:08.204050 (+ 280us) server_negotiation.cc:299] Negotiation successful
0621 02:38:08.204278 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:38:08.205420 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:08.287665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 502
I20260621 02:38:08.304176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 636
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:38:08.320258 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:08.305038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.0.97.2:44075)
0621 02:38:08.305652 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:08.305663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:08.305696 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:38:08.305756 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:08.305762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:08.305975 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:08.306108 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:08.307296 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.308308 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.309729 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.310290 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.310397 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:08.314375 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:08.314386 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:08.314399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:08.314429 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:08.317590 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.318224 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.318231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.318234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.318347 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.318747 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.318752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.318756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.319080 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:08.319385 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:08.319664 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 02:38:08.319930 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":329,"threads_started":1}
I20260621 02:38:08.321578 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:08.402765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 636
I20260621 02:38:08.425199 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 769
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:38:08.441211 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:08.426244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.0.97.3:34101)
0621 02:38:08.426800 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:08.426808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:08.426837 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:38:08.426904 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:08.426911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:08.427124 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:08.427321 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:08.428527 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.429719 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.430987 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:08.431564 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:08.431668 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:08.435789 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:08.435800 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:08.435815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:08.435854 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:08.438739 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.439407 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.439414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.439417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.439544 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:08.439908 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:08.439911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:08.439913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:08.440243 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:08.440474 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:08.440718 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 02:38:08.440929 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":275,"threads_started":1}
I20260621 02:38:08.442503 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:08.525542 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 769
I20260621 02:38:08.541662 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 408
I20260621 02:38:08.542831 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:08.646180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 408
2026-06-21T02:38:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5267 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[918](info): set up 2 sockets
Jun 21 02:38:08 dist-test-slave-jkn7 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260621 02:38:09.745215 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:09 dist-test-slave-jkn7 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:09.908581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37149
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:10.216835 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:10.217115 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:10.217304 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:10.224231 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:10.224330 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:10.224423 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:10.224541 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:10.224633 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:10.234221 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:10.235529 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:10.236814 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:10.248203 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:10.248263 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:10.248293 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:10.249563 931 server_base.cc:1034] running on GCE node
I20260621 02:38:10.250219 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:10.251472 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:10.252661 931 hybrid_clock.cc:648] HybridClock initialized: now 1782009490252612 us; error 50 us; skew 500 ppm
I20260621 02:38:10.252925 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:10 dist-test-slave-jkn7 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:10.259384 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:10.260799 931 webserver.cc:460] Webserver started at http://127.0.97.62:34269/ using document root <none> and password file <none>
I20260621 02:38:10.261483 931 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:10.261584 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:10.261818 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:10.264495 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "2fef7fb4ef664e9890a975e7e87fa28d"
format_stamp: "Formatted at 2026-06-21 02:38:10 on dist-test-slave-jkn7"
server_key: "01f93380438a404dd2ffd55ed394904e"
server_key_iv: "40dea2d060a620e8a7d5e8e1949f5e5f"
server_key_version: "encryptionkey@0"
I20260621 02:38:10.265170 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "2fef7fb4ef664e9890a975e7e87fa28d"
format_stamp: "Formatted at 2026-06-21 02:38:10 on dist-test-slave-jkn7"
server_key: "01f93380438a404dd2ffd55ed394904e"
server_key_iv: "40dea2d060a620e8a7d5e8e1949f5e5f"
server_key_version: "encryptionkey@0"
I20260621 02:38:10.269932 931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:38:10.272993 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:10.274223 931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 02:38:10.274355 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "2fef7fb4ef664e9890a975e7e87fa28d"
format_stamp: "Formatted at 2026-06-21 02:38:10 on dist-test-slave-jkn7"
server_key: "01f93380438a404dd2ffd55ed394904e"
server_key_iv: "40dea2d060a620e8a7d5e8e1949f5e5f"
server_key_version: "encryptionkey@0"
I20260621 02:38:10.274506 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:10.289995 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:10.293646 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:10.293855 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:10.311408 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37149
I20260621 02:38:10.311439 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37149 every 8 connection(s)
I20260621 02:38:10.312755 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:10.316234 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:10.322649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 931
I20260621 02:38:10.322890 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:10.323688 388 external_mini_cluster.cc:1428] Setting key 2bd319aa69a06a67f8d5ff74f9beba64
I20260621 02:38:10.327191 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d: Bootstrap starting.
I20260621 02:38:10.330605 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:10.331661 998 log.cc:824] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:10.334331 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d: No bootstrap required, opened a new log
Jun 21 02:38:10 dist-test-slave-jkn7 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:10.342563 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fef7fb4ef664e9890a975e7e87fa28d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37149 } }
I20260621 02:38:10.342941 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:10.343070 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fef7fb4ef664e9890a975e7e87fa28d, State: Initialized, Role: FOLLOWER
I20260621 02:38:10.343633 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fef7fb4ef664e9890a975e7e87fa28d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37149 } }
I20260621 02:38:10.343894 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:10.344081 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:10.344285 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:10.346242 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:10.325759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.0.97.62:37149)
0621 02:38:10.326522 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:10.326534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:10.326574 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:38:10.328347 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:10.328368 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:10.328759 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:10.329079 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:10.330886 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:10.332220 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:10.333515 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:10.334150 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:10.334341 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:10.338142 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:10.338149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:10.338163 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:10.338206 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:10.341214 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:10.342177 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:10.342189 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:10.342198 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:10.342439 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:10.342839 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:10.342844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:10.342847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:10.343549 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:10.343870 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:10.344378 (+ 508us) server_negotiation.cc:299] Negotiation successful
0621 02:38:10.344820 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":380,"threads_started":1}
I20260621 02:38:10.347266 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fef7fb4ef664e9890a975e7e87fa28d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37149 } }
I20260621 02:38:10.347788 998 leader_election.cc:302] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fef7fb4ef664e9890a975e7e87fa28d; no voters:
I20260621 02:38:10.348384 998 leader_election.cc:288] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:10.348511 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:10.349793 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [term 1 LEADER]: Becoming Leader. State: Replica: 2fef7fb4ef664e9890a975e7e87fa28d, State: Running, Role: LEADER
I20260621 02:38:10.350255 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fef7fb4ef664e9890a975e7e87fa28d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37149 } }
I20260621 02:38:10.350587 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:10.353662 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fef7fb4ef664e9890a975e7e87fa28d. Latest consensus state: current_term: 1 leader_uuid: "2fef7fb4ef664e9890a975e7e87fa28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fef7fb4ef664e9890a975e7e87fa28d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37149 } } }
I20260621 02:38:10.353976 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:10.353621 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fef7fb4ef664e9890a975e7e87fa28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fef7fb4ef664e9890a975e7e87fa28d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37149 } } }
I20260621 02:38:10.354455 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:10.355609 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:10.361532 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:10.369652 1010 catalog_manager.cc:1347] Generated new cluster ID: 0fa474d85aa64ffe99905efa32ba988d
I20260621 02:38:10.369782 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:10.395141 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:10.396493 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:10.405205 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fef7fb4ef664e9890a975e7e87fa28d: Generated new TSK 0
I20260621 02:38:10.406062 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:38:10.458611 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37149
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:10.777755 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:10.778013 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:10.778136 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:10.785575 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:10.785678 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:10.785791 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:10.795531 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37149
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:10.796671 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:10.798038 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:10.810593 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:10.810648 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:10.810616 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:10.811410 1024 server_base.cc:1034] running on GCE node
I20260621 02:38:10.811957 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:10.812775 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:10.813974 1024 hybrid_clock.cc:648] HybridClock initialized: now 1782009490813916 us; error 60 us; skew 500 ppm
I20260621 02:38:10.814232 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:10 dist-test-slave-jkn7 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:10.817262 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:38:10.818622 1024 webserver.cc:460] Webserver started at http://127.0.97.1:43967/ using document root <none> and password file <none>
I20260621 02:38:10.819226 1024 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:10.819322 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:10.819561 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:10.822346 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "0be95a640fc849f48756b42d75f59d90"
format_stamp: "Formatted at 2026-06-21 02:38:10 on dist-test-slave-jkn7"
server_key: "6352dd64b8890e8091bf92eb92f079b0"
server_key_iv: "3eeb959bab2fb3fa98105b15d93870a2"
server_key_version: "encryptionkey@0"
I20260621 02:38:10.823117 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0be95a640fc849f48756b42d75f59d90"
format_stamp: "Formatted at 2026-06-21 02:38:10 on dist-test-slave-jkn7"
server_key: "6352dd64b8890e8091bf92eb92f079b0"
server_key_iv: "3eeb959bab2fb3fa98105b15d93870a2"
server_key_version: "encryptionkey@0"
I20260621 02:38:10.828145 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:38:10.831763 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:10.833076 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:10.833253 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "0be95a640fc849f48756b42d75f59d90"
format_stamp: "Formatted at 2026-06-21 02:38:10 on dist-test-slave-jkn7"
server_key: "6352dd64b8890e8091bf92eb92f079b0"
server_key_iv: "3eeb959bab2fb3fa98105b15d93870a2"
server_key_version: "encryptionkey@0"
I20260621 02:38:10.833461 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:10.853578 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:10.858057 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:10.858309 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:10.859184 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:10.860446 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:10.860519 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:10.860605 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:10.860647 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:10.893785 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33005
I20260621 02:38:10.893833 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33005 every 8 connection(s)
I20260621 02:38:10.895200 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:10.904824 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 1024
I20260621 02:38:10.905053 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:10.905800 388 external_mini_cluster.cc:1428] Setting key 4978f74e92a324aabb95b8c1b8da539a
Jun 21 02:38:10 dist-test-slave-jkn7 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:10.917141 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:10.897721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52685 (local address 127.0.97.62:37149)
0621 02:38:10.898171 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:10.898175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:10.899081 (+ 906us) server_negotiation.cc:407] Connection header received
0621 02:38:10.900925 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:10.900930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:10.901022 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:10.901143 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:10.902725 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:10.903445 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:10.905206 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:10.905683 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:10.905820 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:10.910658 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:10.910663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:10.910667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:10.910699 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:10.913822 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:10.914560 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:10.914569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:10.914575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:10.914730 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:10.915120 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:10.915128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:10.915134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:10.915520 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:10.915683 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:10.916406 (+ 723us) server_negotiation.cc:299] Negotiation successful
0621 02:38:10.916651 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":251,"threads_started":1}
I20260621 02:38:10.917886 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:10.898196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37149 (local address 127.0.97.1:52685)
0621 02:38:10.898902 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:10.898950 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:10.900625 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:10.901446 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:10.901454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:10.901972 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:10.902547 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:10.902564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:10.903594 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:10.903602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:10.905041 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:10.905052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:10.905917 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:10.905924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:10.906158 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:10.907484 (+ 1326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:10.907522 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:10.910406 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:10.913982 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:10.913990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:10.914004 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:10.914437 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:10.914840 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:10.914843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:10.914846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:10.915013 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:10.915681 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:10.915687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:10.916019 (+ 332us) client_negotiation.cc:769] Sending connection context
0621 02:38:10.916283 (+ 264us) client_negotiation.cc:240] Negotiation successful
0621 02:38:10.916687 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":275,"threads_started":1}
I20260621 02:38:10.919488 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37149
I20260621 02:38:10.919802 1152 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:10.920508 1152 heartbeater.cc:511] Master 127.0.97.62:37149 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260621 02:38:10.922636 963 ts_manager.cc:194] Registered new tserver with Master: 0be95a640fc849f48756b42d75f59d90 (127.0.97.1:33005)
I20260621 02:38:10.924088 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:38:10.944413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37149
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:11.256718 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:11.257009 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:11.257107 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:11.264349 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:11.264468 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:11.264582 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:11.274500 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37149
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:11.275789 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:11.277074 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:11.290294 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:11.290405 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:11.290764 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:11.291352 1158 server_base.cc:1034] running on GCE node
I20260621 02:38:11.291976 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:11.292984 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:11.294206 1158 hybrid_clock.cc:648] HybridClock initialized: now 1782009491294153 us; error 84 us; skew 500 ppm
I20260621 02:38:11.294518 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:11 dist-test-slave-jkn7 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:11.297710 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:38:11.299086 1158 webserver.cc:460] Webserver started at http://127.0.97.2:39919/ using document root <none> and password file <none>
I20260621 02:38:11.299760 1158 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:11.299906 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:11.300212 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:11.303099 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "72789fdf8014400b88ec494115db6fcf"
format_stamp: "Formatted at 2026-06-21 02:38:11 on dist-test-slave-jkn7"
server_key: "7649bbcf7eea6241bfa9aee3a20fe591"
server_key_iv: "6ba20487fbd838e4e8ad641c8a07bd5d"
server_key_version: "encryptionkey@0"
I20260621 02:38:11.303855 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "72789fdf8014400b88ec494115db6fcf"
format_stamp: "Formatted at 2026-06-21 02:38:11 on dist-test-slave-jkn7"
server_key: "7649bbcf7eea6241bfa9aee3a20fe591"
server_key_iv: "6ba20487fbd838e4e8ad641c8a07bd5d"
server_key_version: "encryptionkey@0"
I20260621 02:38:11.308969 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 02:38:11.312608 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:11.314035 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:11.314198 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "72789fdf8014400b88ec494115db6fcf"
format_stamp: "Formatted at 2026-06-21 02:38:11 on dist-test-slave-jkn7"
server_key: "7649bbcf7eea6241bfa9aee3a20fe591"
server_key_iv: "6ba20487fbd838e4e8ad641c8a07bd5d"
server_key_version: "encryptionkey@0"
I20260621 02:38:11.314374 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:11.329159 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:11.333230 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:11.333580 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:11.334506 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:11.335774 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:11.335844 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:11.336002 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:11.336071 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:11.375066 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40631
I20260621 02:38:11.375114 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40631 every 8 connection(s)
I20260621 02:38:11.376376 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:11.380154 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 1158
I20260621 02:38:11.380364 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:11.380860 388 external_mini_cluster.cc:1428] Setting key 5c6391e554c0486b958384c98825cfbb
Jun 21 02:38:11 dist-test-slave-jkn7 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:38:11.397751 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.379125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45865 (local address 127.0.97.62:37149)
0621 02:38:11.379322 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:11.379328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:11.380701 (+ 1373us) server_negotiation.cc:407] Connection header received
0621 02:38:11.382783 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:11.382790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:11.382866 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:11.382971 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:11.384379 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.385592 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.387379 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.387787 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.387879 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.391964 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:11.391969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:11.391972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:11.392002 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:11.394723 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:11.395434 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:11.395442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:11.395446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:11.395579 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:11.395997 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:11.396004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:11.396008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:11.396270 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:11.396420 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:11.397137 (+ 717us) server_negotiation.cc:299] Negotiation successful
0621 02:38:11.397378 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:38:11.398787 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.379698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37149 (local address 127.0.97.2:45865)
0621 02:38:11.380523 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:11.380569 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:11.382452 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:11.383171 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:11.383179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:11.383675 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:11.384213 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.384229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.385737 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.385741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.387238 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.387254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.388000 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.388009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.388329 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.389116 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:11.389149 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:11.391743 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:11.394867 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:11.394875 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:11.394892 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:11.395313 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:11.395687 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:11.395693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:11.395697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:11.395882 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:11.396393 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:11.396401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:11.396788 (+ 387us) client_negotiation.cc:769] Sending connection context
0621 02:38:11.397056 (+ 268us) client_negotiation.cc:240] Negotiation successful
0621 02:38:11.397609 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":353,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260621 02:38:11.400281 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37149
I20260621 02:38:11.400597 1286 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:11.401407 1286 heartbeater.cc:511] Master 127.0.97.62:37149 requested a full tablet report, sending...
I20260621 02:38:11.402813 963 ts_manager.cc:194] Registered new tserver with Master: 72789fdf8014400b88ec494115db6fcf (127.0.97.2:40631)
I20260621 02:38:11.403618 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:38:11.419695 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37149
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:11.725502 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:11.725754 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:11.725848 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:11.732718 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:11.732815 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:11.732930 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:11.742264 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37149
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:11.743477 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:11.744751 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:11.761878 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:11.762090 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:11.762149 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:11.763165 1291 server_base.cc:1034] running on GCE node
I20260621 02:38:11.763743 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:11.764607 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:11.765841 1291 hybrid_clock.cc:648] HybridClock initialized: now 1782009491765794 us; error 59 us; skew 500 ppm
I20260621 02:38:11.766170 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:11 dist-test-slave-jkn7 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:11.769186 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:38:11.770617 1291 webserver.cc:460] Webserver started at http://127.0.97.3:33299/ using document root <none> and password file <none>
I20260621 02:38:11.771273 1291 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:11.771413 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:11.771716 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:11.774546 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "098e5d6507514e008152cf881a38ffec"
format_stamp: "Formatted at 2026-06-21 02:38:11 on dist-test-slave-jkn7"
server_key: "bbe6e3496d113c0b3e2c7b6e04f42d32"
server_key_iv: "5a16f12e17c5ab57b2cf0f4418b0755a"
server_key_version: "encryptionkey@0"
I20260621 02:38:11.775314 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "098e5d6507514e008152cf881a38ffec"
format_stamp: "Formatted at 2026-06-21 02:38:11 on dist-test-slave-jkn7"
server_key: "bbe6e3496d113c0b3e2c7b6e04f42d32"
server_key_iv: "5a16f12e17c5ab57b2cf0f4418b0755a"
server_key_version: "encryptionkey@0"
I20260621 02:38:11.780194 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:38:11.783494 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:11.784704 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:38:11.784866 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "098e5d6507514e008152cf881a38ffec"
format_stamp: "Formatted at 2026-06-21 02:38:11 on dist-test-slave-jkn7"
server_key: "bbe6e3496d113c0b3e2c7b6e04f42d32"
server_key_iv: "5a16f12e17c5ab57b2cf0f4418b0755a"
server_key_version: "encryptionkey@0"
I20260621 02:38:11.785063 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:11.800619 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:11.804534 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:11.804809 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:11.805734 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:11.807369 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:11.807488 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:11.807646 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:11.807718 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:11.839989 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41037
I20260621 02:38:11.840039 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41037 every 8 connection(s)
I20260621 02:38:11.841264 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:11.844538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 1291
I20260621 02:38:11.844734 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:11.845314 388 external_mini_cluster.cc:1428] Setting key 91ccc963473b1621140651442ede0718
Jun 21 02:38:11 dist-test-slave-jkn7 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:11.861016 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.843656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46887 (local address 127.0.97.62:37149)
0621 02:38:11.843831 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:11.843835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:11.845038 (+ 1203us) server_negotiation.cc:407] Connection header received
0621 02:38:11.847477 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:11.847483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:11.847584 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:11.847692 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:11.849225 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.850025 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.851740 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.852130 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.852221 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.855496 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:11.855500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:11.855503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:11.855532 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:11.858071 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:11.858725 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:11.858728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:11.858730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:11.858832 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:11.859283 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:11.859287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:11.859289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:11.859617 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:11.859724 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:11.860612 (+ 888us) server_negotiation.cc:299] Negotiation successful
0621 02:38:11.860757 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:38:11.861966 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.844022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37149 (local address 127.0.97.3:46887)
0621 02:38:11.844855 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:11.844912 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:11.847147 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:11.847871 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:11.847878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:11.848562 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:11.849083 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.849099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.850240 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.850244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.851595 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.851606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.852302 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.852307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.852500 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.853166 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:11.853191 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:11.855334 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:11.858183 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:11.858188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:11.858199 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:11.858618 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:11.858946 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:11.858950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:11.858954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:11.859160 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:11.859856 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:11.859862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:11.860236 (+ 374us) client_negotiation.cc:769] Sending connection context
0621 02:38:11.860518 (+ 282us) client_negotiation.cc:240] Negotiation successful
0621 02:38:11.860898 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":332,"threads_started":1}
I20260621 02:38:11.863205 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37149
I20260621 02:38:11.863487 1419 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:11.864152 1419 heartbeater.cc:511] Master 127.0.97.62:37149 requested a full tablet report, sending...
I20260621 02:38:11.865551 963 ts_manager.cc:194] Registered new tserver with Master: 098e5d6507514e008152cf881a38ffec (127.0.97.3:41037)
I20260621 02:38:11.866324 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46887
I20260621 02:38:11.872433 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:38:11.890800 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.878310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.0.97.62:37149)
0621 02:38:11.878476 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:11.878480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:11.878997 (+ 517us) server_negotiation.cc:407] Connection header received
0621 02:38:11.879106 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:11.879110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:11.879166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:11.879247 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:11.880563 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.881274 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.882920 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.883306 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.883402 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.885536 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:11.885541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:11.885544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:11.885572 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:11.888229 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:11.888828 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:11.888832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:11.888835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:11.888956 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:11.889389 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:11.889392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:11.889394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:11.889679 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:11.889899 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:11.890368 (+ 469us) server_negotiation.cc:299] Negotiation successful
0621 02:38:11.890516 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:38:11.896575 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:38:11.903671 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:11.926924 1152 heartbeater.cc:503] Master 127.0.97.62:37149 was elected leader, sending a full tablet report...
I20260621 02:38:11.927053 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.915005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33005 (local address 127.0.0.1:43496)
0621 02:38:11.915815 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:11.915858 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:11.916060 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:11.917391 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:11.917400 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:11.917438 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:11.917913 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.917929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.920423 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.920431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.923297 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.923316 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.926051 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.926058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.926280 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.926333 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 02:38:11.926433 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:38:11.926617 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":460,"threads_started":1}
I20260621 02:38:11.928982 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.915264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.0.97.1:33005)
0621 02:38:11.916707 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:11.916719 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:11.916749 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:38:11.916851 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:11.916861 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:11.917125 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:11.918151 (+ 1026us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:11.918405 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.920254 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.923531 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.925775 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.927527 (+ 1752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.928462 (+ 935us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:11.928531 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 02:38:11.928690 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246,"thread_start_us":1123,"threads_started":1}
I20260621 02:38:11.933027 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.916341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40631 (local address 127.0.0.1:57698)
0621 02:38:11.926170 (+ 9829us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:11.926187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:11.926288 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:11.927037 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:11.927043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:11.927069 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:11.927479 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.927492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.929089 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.929096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.930746 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.930756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.932311 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.932317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.932475 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.932505 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:38:11.932597 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 02:38:11.932719 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9689,"thread_start_us":8931,"threads_started":1}
I20260621 02:38:11.933138 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.916796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.0.97.2:40631)
0621 02:38:11.924144 (+ 7348us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:11.924154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:11.926297 (+ 2143us) server_negotiation.cc:407] Connection header received
0621 02:38:11.926435 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:11.926443 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:11.926635 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:11.926816 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:11.927651 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.928964 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.930889 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.932097 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.932224 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.932403 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:11.932659 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 02:38:11.932818 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7171,"thread_start_us":8091,"threads_started":1}
I20260621 02:38:11.932715 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 4c56980515704b7890d061190398f765 (DEFAULT_TABLE table=test-table [id=f87d9cbc7ea941e7af7847ca577b9fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:11.934144 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.915076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41037 (local address 127.0.0.1:53070)
0621 02:38:11.925534 (+ 10458us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:11.925555 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:11.925673 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:11.926499 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:11.926503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:11.926526 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:11.926888 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.926900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.929381 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.929385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.931223 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.931237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.933580 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.933587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.933752 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.933783 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:38:11.933858 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 02:38:11.933941 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10325,"thread_start_us":10816,"threads_started":1}
I20260621 02:38:11.934684 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c56980515704b7890d061190398f765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:11.935099 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.915575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.0.97.3:41037)
0621 02:38:11.925922 (+ 10347us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:11.925931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:11.925966 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:38:11.926057 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:11.926065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:11.926305 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:11.926517 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:11.927426 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.929175 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.931382 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.932689 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.934377 (+ 1688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.934556 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:11.934631 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 02:38:11.934787 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10149,"thread_start_us":9999,"threads_started":1}
I20260621 02:38:11.937016 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 4c56980515704b7890d061190398f765 (DEFAULT_TABLE table=test-table [id=f87d9cbc7ea941e7af7847ca577b9fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:11.938141 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c56980515704b7890d061190398f765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:11.938299 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 4c56980515704b7890d061190398f765 (DEFAULT_TABLE table=test-table [id=f87d9cbc7ea941e7af7847ca577b9fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:11.939518 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c56980515704b7890d061190398f765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:11.948810 1437 tablet_bootstrap.cc:492] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf: Bootstrap starting.
I20260621 02:38:11.951974 1437 tablet_bootstrap.cc:654] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:11.953059 1437 log.cc:824] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:11.954895 1438 tablet_bootstrap.cc:492] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90: Bootstrap starting.
I20260621 02:38:11.955827 1440 tablet_bootstrap.cc:492] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec: Bootstrap starting.
I20260621 02:38:11.956724 1437 tablet_bootstrap.cc:492] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf: No bootstrap required, opened a new log
I20260621 02:38:11.957015 1437 ts_tablet_manager.cc:1397] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260621 02:38:11.958256 1438 tablet_bootstrap.cc:654] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:11.958665 1440 tablet_bootstrap.cc:654] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:11.959777 1438 log.cc:824] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:11.959971 1440 log.cc:824] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:11.962666 1438 tablet_bootstrap.cc:492] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90: No bootstrap required, opened a new log
I20260621 02:38:11.962664 1440 tablet_bootstrap.cc:492] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec: No bootstrap required, opened a new log
I20260621 02:38:11.962911 1440 ts_tablet_manager.cc:1397] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 02:38:11.962925 1438 ts_tablet_manager.cc:1397] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:38:11.965965 1437 raft_consensus.cc:348] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } }
I20260621 02:38:11.966317 1437 raft_consensus.cc:374] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:11.966425 1437 raft_consensus.cc:729] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72789fdf8014400b88ec494115db6fcf, State: Initialized, Role: FOLLOWER
I20260621 02:38:11.966991 1437 consensus_queue.cc:260] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } }
I20260621 02:38:11.968796 1437 ts_tablet_manager.cc:1428] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260621 02:38:11.973373 1438 raft_consensus.cc:348] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } }
I20260621 02:38:11.973373 1440 raft_consensus.cc:348] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } }
I20260621 02:38:11.973766 1440 raft_consensus.cc:374] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:11.973842 1438 raft_consensus.cc:374] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:11.973935 1440 raft_consensus.cc:729] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e5d6507514e008152cf881a38ffec, State: Initialized, Role: FOLLOWER
I20260621 02:38:11.974011 1438 raft_consensus.cc:729] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be95a640fc849f48756b42d75f59d90, State: Initialized, Role: FOLLOWER
I20260621 02:38:11.974627 1286 heartbeater.cc:503] Master 127.0.97.62:37149 was elected leader, sending a full tablet report...
I20260621 02:38:11.974637 1438 consensus_queue.cc:260] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } }
I20260621 02:38:11.974632 1440 consensus_queue.cc:260] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } }
I20260621 02:38:11.976313 1443 raft_consensus.cc:482] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:11.976657 1443 raft_consensus.cc:504] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } }
I20260621 02:38:11.983299 1438 ts_tablet_manager.cc:1428] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260621 02:38:11.984696 1440 ts_tablet_manager.cc:1428] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec: Time spent starting tablet: real 0.022s user 0.021s sys 0.000s
I20260621 02:38:11.985447 1419 heartbeater.cc:503] Master 127.0.97.62:37149 was elected leader, sending a full tablet report...
I20260621 02:38:11.986568 1443 leader_election.cc:288] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be95a640fc849f48756b42d75f59d90 (127.0.97.1:33005), 098e5d6507514e008152cf881a38ffec (127.0.97.3:41037)
I20260621 02:38:11.994628 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.979645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35101 (local address 127.0.97.1:33005)
0621 02:38:11.979878 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:11.979899 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:11.987314 (+ 7415us) server_negotiation.cc:407] Connection header received
0621 02:38:11.987606 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:11.987617 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:11.987705 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:11.987785 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:11.988751 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.989845 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.992775 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.993750 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.993877 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.993990 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:11.994335 (+ 345us) server_negotiation.cc:299] Negotiation successful
0621 02:38:11.994463 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:38:11.994801 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.979036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33005 (local address 127.0.97.2:35101)
0621 02:38:11.987208 (+ 8172us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:11.987229 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:11.987430 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:11.987956 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:11.987962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:11.987993 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:11.988596 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.988611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.990105 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.990110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.992629 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.992642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.993957 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.993964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.994169 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.994209 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 02:38:11.994294 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 02:38:11.994442 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7994,"thread_start_us":7829,"threads_started":1}
I20260621 02:38:11.995669 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c56980515704b7890d061190398f765" candidate_uuid: "72789fdf8014400b88ec494115db6fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be95a640fc849f48756b42d75f59d90" is_pre_election: true
I20260621 02:38:11.996124 1107 raft_consensus.cc:2454] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72789fdf8014400b88ec494115db6fcf in term 0.
I20260621 02:38:11.996694 1175 leader_election.cc:302] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be95a640fc849f48756b42d75f59d90, 72789fdf8014400b88ec494115db6fcf; no voters:
I20260621 02:38:11.997097 1444 raft_consensus.cc:2781] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:11.997277 1444 raft_consensus.cc:482] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:11.997494 1444 raft_consensus.cc:3037] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:11.997784 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.988164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60979 (local address 127.0.97.3:41037)
0621 02:38:11.988366 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:11.988374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:11.991403 (+ 3029us) server_negotiation.cc:407] Connection header received
0621 02:38:11.991632 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:11.991638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:11.991700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:11.991788 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:11.992834 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.993706 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.995857 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.996824 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.996959 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.997076 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:11.997403 (+ 327us) server_negotiation.cc:299] Negotiation successful
0621 02:38:11.997539 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 02:38:11.997813 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:11.988511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41037 (local address 127.0.97.2:60979)
0621 02:38:11.991318 (+ 2807us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:11.991337 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:11.991492 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:11.991830 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:11.991834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:11.991856 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:11.992687 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.992697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.993811 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.993819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.995721 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:11.995731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:11.997060 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:11.997067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:11.997207 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:11.997228 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:38:11.997367 (+ 139us) client_negotiation.cc:240] Negotiation successful
0621 02:38:11.997507 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2678,"thread_start_us":1798,"threads_started":1}
I20260621 02:38:11.998741 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c56980515704b7890d061190398f765" candidate_uuid: "72789fdf8014400b88ec494115db6fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e5d6507514e008152cf881a38ffec" is_pre_election: true
I20260621 02:38:11.999228 1374 raft_consensus.cc:2454] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72789fdf8014400b88ec494115db6fcf in term 0.
I20260621 02:38:12.000494 1444 raft_consensus.cc:504] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } }
I20260621 02:38:12.001065 1444 leader_election.cc:288] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [CANDIDATE]: Term 1 election: Requested vote from peers 0be95a640fc849f48756b42d75f59d90 (127.0.97.1:33005), 098e5d6507514e008152cf881a38ffec (127.0.97.3:41037)
I20260621 02:38:12.001442 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c56980515704b7890d061190398f765" candidate_uuid: "72789fdf8014400b88ec494115db6fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be95a640fc849f48756b42d75f59d90"
I20260621 02:38:12.001470 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c56980515704b7890d061190398f765" candidate_uuid: "72789fdf8014400b88ec494115db6fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e5d6507514e008152cf881a38ffec"
I20260621 02:38:12.001636 1374 raft_consensus.cc:3037] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:12.001637 1107 raft_consensus.cc:3037] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:12.005050 1374 raft_consensus.cc:2454] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72789fdf8014400b88ec494115db6fcf in term 1.
I20260621 02:38:12.005050 1107 raft_consensus.cc:2454] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72789fdf8014400b88ec494115db6fcf in term 1.
I20260621 02:38:12.005457 1175 leader_election.cc:302] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be95a640fc849f48756b42d75f59d90, 72789fdf8014400b88ec494115db6fcf; no voters:
I20260621 02:38:12.005767 1444 raft_consensus.cc:2781] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:12.005957 1444 raft_consensus.cc:686] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [term 1 LEADER]: Becoming Leader. State: Replica: 72789fdf8014400b88ec494115db6fcf, State: Running, Role: LEADER
I20260621 02:38:12.006417 1444 consensus_queue.cc:237] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } }
I20260621 02:38:12.010061 961 catalog_manager.cc:5495] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 72789fdf8014400b88ec494115db6fcf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "72789fdf8014400b88ec494115db6fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be95a640fc849f48756b42d75f59d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e5d6507514e008152cf881a38ffec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72789fdf8014400b88ec494115db6fcf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40631 } health_report { overall_health: HEALTHY } } }
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:12.030278 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:12.015522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.0.97.1:33005)
0621 02:38:12.015672 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:12.015676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:12.016205 (+ 529us) server_negotiation.cc:407] Connection header received
0621 02:38:12.016409 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:12.016414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:12.016501 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:12.016601 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:12.018257 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.019071 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.020678 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.021133 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.021265 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:12.024951 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:12.024969 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:12.024983 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:12.025016 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:12.027648 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.028310 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.028319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.028323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.028455 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.028796 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.028803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.028807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.029136 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:12.029470 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:12.029749 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 02:38:12.029952 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009492, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009492, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:12.072969 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:12.059564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.0.97.1:33005)
0621 02:38:12.059753 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:12.059760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:12.060114 (+ 354us) server_negotiation.cc:407] Connection header received
0621 02:38:12.060301 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:12.060309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:12.060371 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:12.060480 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:12.061756 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.062476 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.063727 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.064532 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.064663 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:12.068265 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:12.068270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:12.068273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:12.068302 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:12.070669 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.071282 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.071290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.071292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.071427 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.071815 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.071819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.071821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.072074 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:12.072262 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:12.072570 (+ 308us) server_negotiation.cc:299] Negotiation successful
0621 02:38:12.072728 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260621 02:38:12.092883 1420 tablet.cc:2369] T 4c56980515704b7890d061190398f765 P 098e5d6507514e008152cf881a38ffec: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:12.114919 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:12.102037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.0.97.1:33005)
0621 02:38:12.102266 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:12.102271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:12.102345 (+ 74us) server_negotiation.cc:407] Connection header received
0621 02:38:12.102495 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:12.102498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:12.102569 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:12.102653 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:12.103922 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.104573 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.105824 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.106259 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.106361 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:12.109976 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:12.109982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:12.109985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:12.110014 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:12.112232 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.112797 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.112801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.112803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.112920 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.113350 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.113360 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.113364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.114037 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:12.114225 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:12.114497 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 02:38:12.114670 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260621 02:38:12.116691 1067 tablet_service.cc:2908] Rejecting scan request for tablet 4c56980515704b7890d061190398f765: Uninitialized: clean time has not yet been initialized
I20260621 02:38:12.119669 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1024
W20260621 02:38:12.128340 1287 tablet.cc:2369] T 4c56980515704b7890d061190398f765 P 72789fdf8014400b88ec494115db6fcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:38:12.131480 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:12.120563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.0.97.1:33005)
0621 02:38:12.120769 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:12.120777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:12.121025 (+ 248us) server_negotiation.cc:407] Connection header received
0621 02:38:12.121148 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:12.121152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:12.121215 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:12.121345 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:12.122359 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.123008 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.124096 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.124511 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.124612 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:12.126590 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:12.126594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:12.126598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:12.126625 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:12.129177 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.129771 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.129775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.129777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.129879 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.130290 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.130294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.130296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.130549 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:12.130776 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:12.130977 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 02:38:12.131132 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:38:12.132131 1147 generic_service.cc:196] Checking for leaks (request via RPC)
W20260621 02:38:12.220530 1153 tablet.cc:2369] T 4c56980515704b7890d061190398f765 P 0be95a640fc849f48756b42d75f59d90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:38:12.221136 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 1024
I20260621 02:38:12.238504 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1158
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:38:12.252565 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:12.239328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.0.97.2:40631)
0621 02:38:12.239531 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:12.239535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:12.239555 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:38:12.239678 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:12.239682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:12.239747 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:12.239829 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:12.240934 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.241709 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.242873 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.243248 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.243337 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:12.247227 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:12.247239 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:12.247254 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:12.247284 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:12.250058 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.250736 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.250743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.250745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.250898 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.251250 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.251254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.251256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.251569 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:12.251778 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:12.252108 (+ 330us) server_negotiation.cc:299] Negotiation successful
0621 02:38:12.252325 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 02:38:12.253180 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:12.341470 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 1158
I20260621 02:38:12.358044 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1291
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:38:12.371654 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:12.358907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.0.97.3:41037)
0621 02:38:12.359097 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:12.359101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:12.359131 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:38:12.359185 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:12.359189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:12.359256 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:12.359344 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:12.360487 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.361245 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.362542 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:12.362986 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:12.363081 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:12.366900 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:12.366915 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:12.366932 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:12.366962 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:12.369487 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.370056 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.370063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.370065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.370183 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:12.370493 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:12.370496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:12.370498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:12.370798 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:12.371008 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:12.371214 (+ 206us) server_negotiation.cc:299] Negotiation successful
0621 02:38:12.371421 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 02:38:12.372290 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:12.457880 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 1291
I20260621 02:38:12.473606 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 931
I20260621 02:38:12.474692 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:12.579339 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 931
2026-06-21T02:38:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3933 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[1467](info): set up 2 sockets
Jun 21 02:38:12 dist-test-slave-jkn7 krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260621 02:38:13.661183 388 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:13 dist-test-slave-jkn7 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:13.840340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37905
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:14.168010 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:14.168272 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:14.168526 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:14.175493 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:14.175585 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 02:38:14.175664 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:14.175774 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:14.175856 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:14.175969 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:14.185351 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37905
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:14.186578 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:14.187877 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:14.199245 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:14.199354 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:14.199306 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:14.200336 1480 server_base.cc:1034] running on GCE node
I20260621 02:38:14.201046 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:14.202361 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:14.203617 1480 hybrid_clock.cc:648] HybridClock initialized: now 1782009494203564 us; error 33 us; skew 500 ppm
I20260621 02:38:14.203940 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:14 dist-test-slave-jkn7 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:14.207006 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:14.208389 1480 webserver.cc:460] Webserver started at http://127.0.97.62:42325/ using document root <none> and password file <none>
I20260621 02:38:14.209064 1480 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:14.209192 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:14.209554 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:14.212322 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "1649a9bdb19a462ca660761628a2b646"
format_stamp: "Formatted at 2026-06-21 02:38:14 on dist-test-slave-jkn7"
server_key: "d80ca09c18d6b2d0b5a7fc2ee52144e4"
server_key_iv: "654cb17ea8a1327c74a32f532ffb0241"
server_key_version: "encryptionkey@0"
I20260621 02:38:14.213095 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "1649a9bdb19a462ca660761628a2b646"
format_stamp: "Formatted at 2026-06-21 02:38:14 on dist-test-slave-jkn7"
server_key: "d80ca09c18d6b2d0b5a7fc2ee52144e4"
server_key_iv: "654cb17ea8a1327c74a32f532ffb0241"
server_key_version: "encryptionkey@0"
I20260621 02:38:14.218014 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:38:14.221352 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:14.222591 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:38:14.222764 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "1649a9bdb19a462ca660761628a2b646"
format_stamp: "Formatted at 2026-06-21 02:38:14 on dist-test-slave-jkn7"
server_key: "d80ca09c18d6b2d0b5a7fc2ee52144e4"
server_key_iv: "654cb17ea8a1327c74a32f532ffb0241"
server_key_version: "encryptionkey@0"
I20260621 02:38:14.222970 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:14.248687 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:14.252568 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:14.252825 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:14.273442 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37905
I20260621 02:38:14.273463 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37905 every 8 connection(s)
I20260621 02:38:14.274662 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:14.276154 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 1480
I20260621 02:38:14.276368 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:14.277179 388 external_mini_cluster.cc:1428] Setting key f2268ab632fc98fa9f8dd604cf0b6ece
I20260621 02:38:14.279207 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:38:14 dist-test-slave-jkn7 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:14.292083 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646: Bootstrap starting.
I20260621 02:38:14.295164 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:14.296111 1557 log.cc:824] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:14.298789 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:14.279101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.0.97.62:37905)
0621 02:38:14.279899 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:14.279910 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:14.279952 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:38:14.281549 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:14.281569 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:14.281923 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:14.282238 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:14.283867 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.285161 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.286839 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.287448 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.287563 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:14.292040 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:14.292050 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:14.292063 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:14.292099 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:14.295159 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:14.295798 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:14.295813 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:14.295819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:14.295977 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:14.296328 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:14.296335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:14.296338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:14.296831 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:14.297171 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:14.297523 (+ 352us) server_negotiation.cc:299] Negotiation successful
0621 02:38:14.297901 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":423,"threads_started":1}
I20260621 02:38:14.298830 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646: No bootstrap required, opened a new log
I20260621 02:38:14.306216 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1649a9bdb19a462ca660761628a2b646" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } }
I20260621 02:38:14.306614 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:14.306723 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1649a9bdb19a462ca660761628a2b646, State: Initialized, Role: FOLLOWER
I20260621 02:38:14.307305 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1649a9bdb19a462ca660761628a2b646" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } }
I20260621 02:38:14.307529 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:14.307679 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:14.307848 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:14.309899 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1649a9bdb19a462ca660761628a2b646" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } }
I20260621 02:38:14.310405 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1649a9bdb19a462ca660761628a2b646; no voters:
I20260621 02:38:14.311024 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:14.311164 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:14.311903 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [term 1 LEADER]: Becoming Leader. State: Replica: 1649a9bdb19a462ca660761628a2b646, State: Running, Role: LEADER
I20260621 02:38:14.312422 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1649a9bdb19a462ca660761628a2b646" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } }
I20260621 02:38:14.312953 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:14.315039 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1649a9bdb19a462ca660761628a2b646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1649a9bdb19a462ca660761628a2b646" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } } }
I20260621 02:38:14.315382 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:14.315485 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1649a9bdb19a462ca660761628a2b646. Latest consensus state: current_term: 1 leader_uuid: "1649a9bdb19a462ca660761628a2b646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1649a9bdb19a462ca660761628a2b646" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } } }
I20260621 02:38:14.315937 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:14.317778 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:14.322317 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:14.330662 1567 catalog_manager.cc:1347] Generated new cluster ID: 9a690cfaeb874aa58ca1001b0f306fc0
I20260621 02:38:14.330773 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:14.352286 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:14.353658 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:14.364295 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646: Generated new TSK 0
I20260621 02:38:14.365062 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:38:14.407807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37905
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:14.718165 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:14.718477 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:14.718655 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:14.726071 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:14.726215 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:14.726351 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:38:14.726483 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:14.735981 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:14.737229 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:14.738588 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:14.751039 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:14.750986 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:14.751092 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:14.752247 1583 server_base.cc:1034] running on GCE node
I20260621 02:38:14.752848 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:14.753842 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:14.755085 1583 hybrid_clock.cc:648] HybridClock initialized: now 1782009494755029 us; error 35 us; skew 500 ppm
I20260621 02:38:14.755380 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:14 dist-test-slave-jkn7 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:14.758586 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:38:14.759918 1583 webserver.cc:460] Webserver started at http://127.0.97.1:33627/ using document root <none> and password file <none>
I20260621 02:38:14.760632 1583 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:14.760721 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:14.760990 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:14.763784 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "1d6dce860a0e48b1bfac27366c25364d"
format_stamp: "Formatted at 2026-06-21 02:38:14 on dist-test-slave-jkn7"
server_key: "ddfcc25dd5d1668fe94e1d9292fff9c8"
server_key_iv: "9658c4b890e3fb7bf01510756ec0c2e7"
server_key_version: "encryptionkey@0"
I20260621 02:38:14.764493 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1d6dce860a0e48b1bfac27366c25364d"
format_stamp: "Formatted at 2026-06-21 02:38:14 on dist-test-slave-jkn7"
server_key: "ddfcc25dd5d1668fe94e1d9292fff9c8"
server_key_iv: "9658c4b890e3fb7bf01510756ec0c2e7"
server_key_version: "encryptionkey@0"
I20260621 02:38:14.769400 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:38:14.772687 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:14.774031 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:14.774191 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "1d6dce860a0e48b1bfac27366c25364d"
format_stamp: "Formatted at 2026-06-21 02:38:14 on dist-test-slave-jkn7"
server_key: "ddfcc25dd5d1668fe94e1d9292fff9c8"
server_key_iv: "9658c4b890e3fb7bf01510756ec0c2e7"
server_key_version: "encryptionkey@0"
I20260621 02:38:14.774416 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:14.784878 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:14.788897 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:14.789186 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:14.791982 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:14.792102 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:14.792254 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:14.792384 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 21 02:38:14 dist-test-slave-jkn7 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:14.818102 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:14.792852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.97.1:43751)
0621 02:38:14.793725 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:14.793774 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:14.795624 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:14.796657 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:14.796665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:14.797548 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:14.798505 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:14.798536 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.800247 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.800258 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:14.802521 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:14.802535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.803655 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.803662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:14.803977 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:14.805081 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:14.805129 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:14.808892 (+ 3763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:14.813344 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:14.813355 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:14.813383 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:14.814166 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:14.814618 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:14.814624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:14.814629 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:14.814851 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:14.815445 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:14.815455 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:14.815874 (+ 419us) client_negotiation.cc:769] Sending connection context
0621 02:38:14.816433 (+ 559us) client_negotiation.cc:240] Negotiation successful
0621 02:38:14.816880 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":385,"threads_started":1}
I20260621 02:38:14.819234 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:14.792411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43751 (local address 127.0.97.62:37905)
0621 02:38:14.792602 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:14.792609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:14.793896 (+ 1287us) server_negotiation.cc:407] Connection header received
0621 02:38:14.796080 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:14.796088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:14.796167 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:14.796226 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:14.798748 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.800049 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.802691 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.803359 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.803505 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:14.809188 (+ 5683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:14.809194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:14.809198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:14.809236 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:14.813096 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:14.814322 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:14.814329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:14.814333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:14.814493 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:14.814987 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:14.814993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:14.814997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:14.815299 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:14.815465 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:14.818654 (+ 3189us) server_negotiation.cc:299] Negotiation successful
0621 02:38:14.818892 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:38:14.842864 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:14.825460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.97.1:42119)
0621 02:38:14.826621 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:14.826674 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:14.826859 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:14.827283 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:14.827291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:14.828109 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:14.828498 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:14.828508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.830026 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.830042 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:14.831959 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:14.831984 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.832964 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.832971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:14.833153 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:14.833935 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:14.833993 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:14.835368 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:14.839592 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:14.839600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:14.839607 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:14.840268 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:14.840747 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:14.840754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:14.840759 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:14.840964 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:14.841762 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:14.841766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:14.842020 (+ 254us) client_negotiation.cc:769] Sending connection context
0621 02:38:14.842202 (+ 182us) client_negotiation.cc:240] Negotiation successful
0621 02:38:14.842490 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":647,"threads_started":1}
I20260621 02:38:14.843587 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:14.825507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42119 (local address 127.0.97.62:37905)
0621 02:38:14.825732 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:14.825741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:14.826767 (+ 1026us) server_negotiation.cc:407] Connection header received
0621 02:38:14.827022 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:14.827027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:14.827099 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:14.827243 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:14.828659 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.829857 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.832191 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:14.832749 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:14.832902 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:14.835607 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:14.835613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:14.835617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:14.835658 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:14.839387 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:14.840446 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:14.840451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:14.840453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:14.840577 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:14.841089 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:14.841095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:14.841099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:14.841534 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:14.841699 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:14.843183 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0621 02:38:14.843360 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:38:14.846757 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36063
I20260621 02:38:14.846782 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36063 every 8 connection(s)
I20260621 02:38:14.848042 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:14.850142 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37905
I20260621 02:38:14.850462 1718 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:14.851087 1718 heartbeater.cc:511] Master 127.0.97.62:37905 requested a full tablet report, sending...
I20260621 02:38:14.852973 1512 ts_manager.cc:194] Registered new tserver with Master: 1d6dce860a0e48b1bfac27366c25364d (127.0.97.1:36063)
I20260621 02:38:14.853168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 1583
I20260621 02:38:14.853421 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:14.854043 388 external_mini_cluster.cc:1428] Setting key f7d6e877fffb4ca5c36437b8b8d5d3e2
I20260621 02:38:14.854704 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43751
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:38:14.891817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37905
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:15.200133 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:15.200425 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:15.200599 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:15.207630 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:15.207772 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:15.207906 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:38:15.208036 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:15.217407 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:15.218636 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:15.219988 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:15.232391 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:15.232314 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:15.232352 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:15.233418 1722 server_base.cc:1034] running on GCE node
I20260621 02:38:15.233917 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:15.234721 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:15.235931 1722 hybrid_clock.cc:648] HybridClock initialized: now 1782009495235880 us; error 24 us; skew 500 ppm
I20260621 02:38:15.236217 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:15 dist-test-slave-jkn7 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:15.239223 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:38:15.240557 1722 webserver.cc:460] Webserver started at http://127.0.97.2:45155/ using document root <none> and password file <none>
I20260621 02:38:15.241219 1722 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:15.241370 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:15.241689 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:15.244355 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "cce5d4032aaa441ebcbc29ab40105b72"
format_stamp: "Formatted at 2026-06-21 02:38:15 on dist-test-slave-jkn7"
server_key: "a9ea3778f54e20b2d6ad58f60dd7cafa"
server_key_iv: "bded3cfa6a0296a9c4ee56a062e5d289"
server_key_version: "encryptionkey@0"
I20260621 02:38:15.245107 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cce5d4032aaa441ebcbc29ab40105b72"
format_stamp: "Formatted at 2026-06-21 02:38:15 on dist-test-slave-jkn7"
server_key: "a9ea3778f54e20b2d6ad58f60dd7cafa"
server_key_iv: "bded3cfa6a0296a9c4ee56a062e5d289"
server_key_version: "encryptionkey@0"
I20260621 02:38:15.249976 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:38:15.253159 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:15.254386 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:38:15.254562 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "cce5d4032aaa441ebcbc29ab40105b72"
format_stamp: "Formatted at 2026-06-21 02:38:15 on dist-test-slave-jkn7"
server_key: "a9ea3778f54e20b2d6ad58f60dd7cafa"
server_key_iv: "bded3cfa6a0296a9c4ee56a062e5d289"
server_key_version: "encryptionkey@0"
I20260621 02:38:15.254776 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:15.308406 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:15.312628 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:15.312914 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:15.315548 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:15.315649 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:15.315769 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:15.315843 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:38:15 dist-test-slave-jkn7 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:15.339591 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.316604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53207 (local address 127.0.97.62:37905)
0621 02:38:15.316760 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:15.316765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:15.317894 (+ 1129us) server_negotiation.cc:407] Connection header received
0621 02:38:15.320053 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:15.320060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:15.320142 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:15.320262 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:15.322400 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.323465 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.326657 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.327133 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.327276 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.332617 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:15.332622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:15.332625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:15.332655 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:15.335783 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.336806 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.336815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.336819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.336970 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.337492 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.337498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.337503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.337806 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:15.337986 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:15.339041 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0621 02:38:15.339255 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:38:15.340608 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.316853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.97.2:53207)
0621 02:38:15.317740 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:15.317777 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:15.319773 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:15.320512 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:15.320520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:15.321227 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:15.322220 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.322242 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.323644 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.323651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.326515 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.326530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.327405 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.327414 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.327769 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.328578 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:15.328619 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:15.332346 (+ 3727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:15.335921 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:15.335933 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:15.335956 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:15.336611 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:15.337068 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:15.337073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:15.337078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:15.337327 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:15.338080 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:15.338089 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:15.338587 (+ 498us) client_negotiation.cc:769] Sending connection context
0621 02:38:15.338970 (+ 383us) client_negotiation.cc:240] Negotiation successful
0621 02:38:15.339474 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":426,"threads_started":1}
I20260621 02:38:15.360790 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.347325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38317 (local address 127.0.97.62:37905)
0621 02:38:15.347485 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:15.347489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:15.348152 (+ 663us) server_negotiation.cc:407] Connection header received
0621 02:38:15.348289 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:15.348298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:15.348410 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:15.348542 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:15.349448 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.350494 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.352639 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.353210 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.353370 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.355051 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:15.355065 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:15.355070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:15.355109 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:15.357740 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.358392 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.358399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.358402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.358543 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.359089 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.359097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.359102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.359574 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:15.359758 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:15.360178 (+ 420us) server_negotiation.cc:299] Negotiation successful
0621 02:38:15.360425 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:38:15.360913 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.347419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.97.2:38317)
0621 02:38:15.348049 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:15.348067 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:15.348185 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:15.348553 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:15.348562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:15.348985 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:15.349254 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.349263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.350697 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.350701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.352423 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.352445 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.353410 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.353416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.353579 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.354201 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:15.354217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:15.354885 (+ 668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:15.357859 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:15.357866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:15.357871 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:15.358278 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:15.358779 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:15.358786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:15.358792 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:15.358959 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:15.359707 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:15.359711 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:15.359893 (+ 182us) client_negotiation.cc:769] Sending connection context
0621 02:38:15.360043 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 02:38:15.360369 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":388,"threads_started":1}
I20260621 02:38:15.373384 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33561
I20260621 02:38:15.373422 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33561 every 8 connection(s)
I20260621 02:38:15.374529 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:15.376545 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37905
I20260621 02:38:15.376724 1857 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:15.377350 1857 heartbeater.cc:511] Master 127.0.97.62:37905 requested a full tablet report, sending...
I20260621 02:38:15.378623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 1722
I20260621 02:38:15.378623 1511 ts_manager.cc:194] Registered new tserver with Master: cce5d4032aaa441ebcbc29ab40105b72 (127.0.97.2:33561)
I20260621 02:38:15.378810 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:15.379348 388 external_mini_cluster.cc:1428] Setting key 83c01d52df640a98fc8772dc27fde0d0
I20260621 02:38:15.379756 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53207
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:38:15.417794 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37905
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:15.734510 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:15.734815 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:15.734994 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:15.741962 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:15.742103 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:15.742251 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:38:15.742388 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:15.751632 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:15.752828 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:15.754305 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:15.766722 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:15.766788 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:15.767068 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:15.767809 1861 server_base.cc:1034] running on GCE node
I20260621 02:38:15.768401 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:15.769349 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:15.770586 1861 hybrid_clock.cc:648] HybridClock initialized: now 1782009495770531 us; error 34 us; skew 500 ppm
I20260621 02:38:15.770874 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:15 dist-test-slave-jkn7 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:15.773923 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:38:15.775228 1861 webserver.cc:460] Webserver started at http://127.0.97.3:41935/ using document root <none> and password file <none>
I20260621 02:38:15.775836 1861 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:15.775914 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:15.776407 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:15.779377 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "172c5c91b0124f3a9cf694fc16662f32"
format_stamp: "Formatted at 2026-06-21 02:38:15 on dist-test-slave-jkn7"
server_key: "b25503be377ce5e3ae7e473a6c49305b"
server_key_iv: "335f359aebcbee67e7ce214f4a90e39a"
server_key_version: "encryptionkey@0"
I20260621 02:38:15.780085 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "172c5c91b0124f3a9cf694fc16662f32"
format_stamp: "Formatted at 2026-06-21 02:38:15 on dist-test-slave-jkn7"
server_key: "b25503be377ce5e3ae7e473a6c49305b"
server_key_iv: "335f359aebcbee67e7ce214f4a90e39a"
server_key_version: "encryptionkey@0"
I20260621 02:38:15.784912 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:38:15.788393 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:15.789806 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:38:15.789950 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "172c5c91b0124f3a9cf694fc16662f32"
format_stamp: "Formatted at 2026-06-21 02:38:15 on dist-test-slave-jkn7"
server_key: "b25503be377ce5e3ae7e473a6c49305b"
server_key_iv: "335f359aebcbee67e7ce214f4a90e39a"
server_key_version: "encryptionkey@0"
I20260621 02:38:15.790128 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:15.816624 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:15.820889 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:15.821195 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:15.823882 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:15.823987 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:15.824138 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:15.824245 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:38:15 dist-test-slave-jkn7 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:15.851984 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.825088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.97.3:48135)
0621 02:38:15.826049 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:15.826101 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:15.828904 (+ 2803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:15.829766 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:15.829774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:15.830550 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:15.831568 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.831603 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.832942 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.832951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.835085 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.835099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.836017 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.836027 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.836391 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.837190 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:15.837227 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:15.841498 (+ 4271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:15.845929 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:15.845941 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:15.845957 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:15.846759 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:15.847306 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:15.847314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:15.847320 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:15.847616 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:15.848489 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:15.848500 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:15.849038 (+ 538us) client_negotiation.cc:769] Sending connection context
0621 02:38:15.849413 (+ 375us) client_negotiation.cc:240] Negotiation successful
0621 02:38:15.850101 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":452,"threads_started":1}
I20260621 02:38:15.852849 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.824471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48135 (local address 127.0.97.62:37905)
0621 02:38:15.824681 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:15.824687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:15.826229 (+ 1542us) server_negotiation.cc:407] Connection header received
0621 02:38:15.829265 (+ 3036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:15.829275 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:15.829393 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:15.829540 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:15.831768 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.832794 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.835244 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.835746 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.835886 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.841805 (+ 5919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:15.841814 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:15.841822 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:15.841867 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:15.845712 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.846944 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.846953 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.846960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.847150 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.847774 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.847781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.847787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.848234 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:15.848405 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:15.852034 (+ 3629us) server_negotiation.cc:299] Negotiation successful
0621 02:38:15.852345 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:38:15.857599 1718 heartbeater.cc:503] Master 127.0.97.62:37905 was elected leader, sending a full tablet report...
I20260621 02:38:15.873425 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42029
I20260621 02:38:15.873459 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42029 every 8 connection(s)
I20260621 02:38:15.874318 1965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.859203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.97.3:56317)
0621 02:38:15.859831 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:15.859851 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:15.859955 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:15.860286 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:15.860289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:15.860697 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:15.860979 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.860987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.862548 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.862557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.864670 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.864690 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.865516 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.865523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.865692 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.866335 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:15.866364 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:15.867458 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:15.871580 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:15.871588 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:15.871593 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:15.871961 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:15.872360 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:15.872364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:15.872368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:15.872545 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:15.873089 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:15.873094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:15.873338 (+ 244us) client_negotiation.cc:769] Sending connection context
0621 02:38:15.873521 (+ 183us) client_negotiation.cc:240] Negotiation successful
0621 02:38:15.873862 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":338,"threads_started":1}
I20260621 02:38:15.874277 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.859163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56317 (local address 127.0.97.62:37905)
0621 02:38:15.859331 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:15.859338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:15.859939 (+ 601us) server_negotiation.cc:407] Connection header received
0621 02:38:15.860045 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:15.860053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:15.860145 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:15.860258 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:15.861113 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.862365 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.864823 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.865247 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.865444 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.867662 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:15.867671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:15.867680 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:15.867725 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:15.871451 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.872082 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.872087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.872090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.872240 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.872698 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.872703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.872707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.872963 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:15.873117 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:15.873738 (+ 621us) server_negotiation.cc:299] Negotiation successful
0621 02:38:15.873937 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:38:15.874758 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:15.876825 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37905
I20260621 02:38:15.877038 1996 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:15.877970 1996 heartbeater.cc:511] Master 127.0.97.62:37905 requested a full tablet report, sending...
I20260621 02:38:15.879271 1511 ts_manager.cc:194] Registered new tserver with Master: 172c5c91b0124f3a9cf694fc16662f32 (127.0.97.3:42029)
I20260621 02:38:15.880010 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48135
I20260621 02:38:15.884752 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 1861
I20260621 02:38:15.884939 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:15.885411 388 external_mini_cluster.cc:1428] Setting key 987f29941d56cfc984546d1046631a71
I20260621 02:38:15.888177 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:38:15 dist-test-slave-jkn7 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:38:15 dist-test-slave-jkn7 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:15.931214 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.917770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.0.97.62:37905)
0621 02:38:15.917942 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:15.917946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:15.918223 (+ 277us) server_negotiation.cc:407] Connection header received
0621 02:38:15.918409 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:15.918413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:15.918479 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:15.918567 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:15.919915 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.920769 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.922145 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.922583 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.922673 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.926433 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:15.926438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:15.926441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:15.926469 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:15.929067 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.929696 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.929700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.929702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.929809 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:15.930124 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:15.930127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:15.930129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:15.930372 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:15.930562 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:15.930752 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 02:38:15.930892 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:38:15.935124 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:38:15.941771 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:15.976527 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.953853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42029 (local address 127.0.0.1:37788)
0621 02:38:15.966634 (+ 12781us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:15.966687 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:15.966889 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:15.968200 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:15.968208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:15.968239 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:15.968733 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.968749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.971196 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.971205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.973785 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.973804 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.975517 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.975525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.975747 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.975812 (+ 65us) client_negotiation.cc:769] Sending connection context
0621 02:38:15.975937 (+ 125us) client_negotiation.cc:240] Negotiation successful
0621 02:38:15.976128 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12501,"thread_start_us":14354,"threads_started":1}
I20260621 02:38:15.977262 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.953939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36063 (local address 127.0.0.1:39642)
0621 02:38:15.968505 (+ 14566us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:15.968525 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:15.968645 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:15.969741 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:15.969749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:15.969781 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:15.970311 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.970326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.972448 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.972452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.974122 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.974137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.976263 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.976271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.976537 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.976569 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:38:15.976660 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 02:38:15.976842 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14411,"mutex_wait_us":24,"thread_start_us":12839,"threads_started":1}
I20260621 02:38:15.977340 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.954001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.0.97.3:42029)
0621 02:38:15.967620 (+ 13619us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:15.967631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:15.967670 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:38:15.967767 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:15.967772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:15.967977 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:15.968187 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:15.968966 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.971025 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.973958 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.975240 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.976454 (+ 1214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.976710 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:15.976779 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 02:38:15.976966 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13450,"thread_start_us":13279,"threads_started":1}
I20260621 02:38:15.978093 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.954261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.0.97.1:36063)
0621 02:38:15.966534 (+ 12273us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:15.966543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:15.968780 (+ 2237us) server_negotiation.cc:407] Connection header received
0621 02:38:15.968877 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:15.968886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:15.969199 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:15.969460 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:15.970533 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.972290 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.974286 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.975747 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.975869 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.976108 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:15.977611 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0621 02:38:15.977763 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12081,"thread_start_us":12281,"threads_started":1}
I20260621 02:38:15.979682 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.968950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.0.97.2:33561)
0621 02:38:15.970927 (+ 1977us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:15.970935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:15.971838 (+ 903us) server_negotiation.cc:407] Connection header received
0621 02:38:15.971939 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:15.971948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:15.972177 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:15.973187 (+ 1010us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:15.973404 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.974786 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.977377 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.978666 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.978822 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.979006 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:15.979238 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 02:38:15.979401 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1831,"thread_start_us":1732,"threads_started":1}
I20260621 02:38:15.980448 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:15.968826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33561 (local address 127.0.0.1:58438)
0621 02:38:15.971596 (+ 2770us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:15.971619 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:15.971726 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:15.972374 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:15.972379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:15.972509 (+ 130us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:15.972970 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.972982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.974935 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.974949 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.977150 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:15.977163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:15.978856 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:15.978862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:15.979057 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:15.979086 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 02:38:15.979196 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 02:38:15.980046 (+ 850us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2625,"mutex_wait_us":62,"thread_start_us":2245,"threads_started":1}
I20260621 02:38:15.981474 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 75b2b814ca524796b7b3c43b0352f6e5 (DEFAULT_TABLE table=test-table [id=ce108a409b424154a6abfcfb20e2169d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:15.981837 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 75b2b814ca524796b7b3c43b0352f6e5 (DEFAULT_TABLE table=test-table [id=ce108a409b424154a6abfcfb20e2169d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:15.983021 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b2b814ca524796b7b3c43b0352f6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:15.983065 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b2b814ca524796b7b3c43b0352f6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:15.983778 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 75b2b814ca524796b7b3c43b0352f6e5 (DEFAULT_TABLE table=test-table [id=ce108a409b424154a6abfcfb20e2169d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:15.985072 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b2b814ca524796b7b3c43b0352f6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:15.994724 2015 tablet_bootstrap.cc:492] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: Bootstrap starting.
I20260621 02:38:15.997953 2015 tablet_bootstrap.cc:654] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:15.999591 2016 tablet_bootstrap.cc:492] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: Bootstrap starting.
I20260621 02:38:15.999729 2015 log.cc:824] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:16.000303 2017 tablet_bootstrap.cc:492] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: Bootstrap starting.
I20260621 02:38:16.003115 2016 tablet_bootstrap.cc:654] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:16.003628 2017 tablet_bootstrap.cc:654] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:16.004266 2016 log.cc:824] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:16.004398 2015 tablet_bootstrap.cc:492] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: No bootstrap required, opened a new log
I20260621 02:38:16.004585 2017 log.cc:824] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:16.004657 2015 ts_tablet_manager.cc:1397] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:38:16.007113 2016 tablet_bootstrap.cc:492] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: No bootstrap required, opened a new log
I20260621 02:38:16.007306 2017 tablet_bootstrap.cc:492] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: No bootstrap required, opened a new log
I20260621 02:38:16.007364 2016 ts_tablet_manager.cc:1397] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260621 02:38:16.007655 2017 ts_tablet_manager.cc:1397] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 02:38:16.012374 2015 raft_consensus.cc:348] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } }
I20260621 02:38:16.012740 2015 raft_consensus.cc:374] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:16.012858 2015 raft_consensus.cc:729] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6dce860a0e48b1bfac27366c25364d, State: Initialized, Role: FOLLOWER
I20260621 02:38:16.013499 2015 consensus_queue.cc:260] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } }
I20260621 02:38:16.015153 2015 ts_tablet_manager.cc:1428] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260621 02:38:16.018208 2016 raft_consensus.cc:348] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } }
I20260621 02:38:16.018613 2016 raft_consensus.cc:374] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:16.018385 2017 raft_consensus.cc:348] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } }
I20260621 02:38:16.018708 2016 raft_consensus.cc:729] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172c5c91b0124f3a9cf694fc16662f32, State: Initialized, Role: FOLLOWER
I20260621 02:38:16.018739 2017 raft_consensus.cc:374] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:16.018818 2017 raft_consensus.cc:729] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce5d4032aaa441ebcbc29ab40105b72, State: Initialized, Role: FOLLOWER
I20260621 02:38:16.019299 2016 consensus_queue.cc:260] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } }
I20260621 02:38:16.019299 2017 consensus_queue.cc:260] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } }
I20260621 02:38:16.020336 1996 heartbeater.cc:503] Master 127.0.97.62:37905 was elected leader, sending a full tablet report...
I20260621 02:38:16.020887 1857 heartbeater.cc:503] Master 127.0.97.62:37905 was elected leader, sending a full tablet report...
I20260621 02:38:16.021071 2016 ts_tablet_manager.cc:1428] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260621 02:38:16.021512 2017 ts_tablet_manager.cc:1428] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
W20260621 02:38:16.100461 1719 tablet.cc:2369] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:38:16.126559 1858 tablet.cc:2369] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:38:16.128345 1997 tablet.cc:2369] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:38:16.241196 2022 raft_consensus.cc:482] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:16.241516 2022 raft_consensus.cc:504] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } }
I20260621 02:38:16.243216 2022 leader_election.cc:288] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d6dce860a0e48b1bfac27366c25364d (127.0.97.1:36063), cce5d4032aaa441ebcbc29ab40105b72 (127.0.97.2:33561)
I20260621 02:38:16.250027 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.243815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48023 (local address 127.0.97.1:36063)
0621 02:38:16.244028 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.244036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.244063 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:38:16.244645 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.244651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.244749 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.244872 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:16.245421 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.246368 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.248116 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.249091 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.249189 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.249261 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:16.249646 (+ 385us) server_negotiation.cc:299] Negotiation successful
0621 02:38:16.249778 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:38:16.250051 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.243548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36063 (local address 127.0.97.3:48023)
0621 02:38:16.243851 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:16.243873 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:16.244051 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:16.244901 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:16.244905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:16.244928 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:16.245170 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.245183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.246499 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.246504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.247947 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.247958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.249242 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.249248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.249478 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.249516 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 02:38:16.249619 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:38:16.249762 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260621 02:38:16.250648 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.243710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50143 (local address 127.0.97.2:33561)
0621 02:38:16.243929 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.243936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.245040 (+ 1104us) server_negotiation.cc:407] Connection header received
0621 02:38:16.245216 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.245224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.245335 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.245433 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:16.246033 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.246893 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.248688 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.249641 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.249753 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.249853 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:16.250238 (+ 385us) server_negotiation.cc:299] Negotiation successful
0621 02:38:16.250400 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:38:16.251153 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.243619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33561 (local address 127.0.97.3:50143)
0621 02:38:16.244922 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:16.244943 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:16.245081 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:16.245475 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:16.245481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:16.245510 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:16.245877 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.245888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.247020 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.247028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.248544 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.248557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.249839 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.249846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.250034 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.250077 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 02:38:16.250835 (+ 758us) client_negotiation.cc:240] Negotiation successful
0621 02:38:16.250933 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"thread_start_us":946,"threads_started":1}
I20260621 02:38:16.251166 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b2b814ca524796b7b3c43b0352f6e5" candidate_uuid: "172c5c91b0124f3a9cf694fc16662f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dce860a0e48b1bfac27366c25364d" is_pre_election: true
I20260621 02:38:16.251787 1673 raft_consensus.cc:2454] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 172c5c91b0124f3a9cf694fc16662f32 in term 0.
I20260621 02:38:16.252103 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b2b814ca524796b7b3c43b0352f6e5" candidate_uuid: "172c5c91b0124f3a9cf694fc16662f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce5d4032aaa441ebcbc29ab40105b72" is_pre_election: true
I20260621 02:38:16.252513 1879 leader_election.cc:302] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 172c5c91b0124f3a9cf694fc16662f32, 1d6dce860a0e48b1bfac27366c25364d; no voters:
I20260621 02:38:16.252666 1811 raft_consensus.cc:2454] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 172c5c91b0124f3a9cf694fc16662f32 in term 0.
I20260621 02:38:16.252959 2022 raft_consensus.cc:2781] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:16.253144 2022 raft_consensus.cc:482] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:16.253271 2022 raft_consensus.cc:3037] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:16.255663 2022 raft_consensus.cc:504] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } }
I20260621 02:38:16.256332 2022 leader_election.cc:288] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [CANDIDATE]: Term 1 election: Requested vote from peers 1d6dce860a0e48b1bfac27366c25364d (127.0.97.1:36063), cce5d4032aaa441ebcbc29ab40105b72 (127.0.97.2:33561)
I20260621 02:38:16.256763 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b2b814ca524796b7b3c43b0352f6e5" candidate_uuid: "172c5c91b0124f3a9cf694fc16662f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dce860a0e48b1bfac27366c25364d"
I20260621 02:38:16.256803 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b2b814ca524796b7b3c43b0352f6e5" candidate_uuid: "172c5c91b0124f3a9cf694fc16662f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce5d4032aaa441ebcbc29ab40105b72"
I20260621 02:38:16.256961 1673 raft_consensus.cc:3037] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:16.257001 1811 raft_consensus.cc:3037] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:16.259593 1811 raft_consensus.cc:2454] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 172c5c91b0124f3a9cf694fc16662f32 in term 1.
I20260621 02:38:16.259738 1673 raft_consensus.cc:2454] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 172c5c91b0124f3a9cf694fc16662f32 in term 1.
I20260621 02:38:16.260133 1878 leader_election.cc:302] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 172c5c91b0124f3a9cf694fc16662f32, cce5d4032aaa441ebcbc29ab40105b72; no voters:
I20260621 02:38:16.260449 2022 raft_consensus.cc:2781] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:16.260947 2022 raft_consensus.cc:686] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 1 LEADER]: Becoming Leader. State: Replica: 172c5c91b0124f3a9cf694fc16662f32, State: Running, Role: LEADER
I20260621 02:38:16.261550 2022 consensus_queue.cc:237] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } }
I20260621 02:38:16.266260 1511 catalog_manager.cc:5495] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 172c5c91b0124f3a9cf694fc16662f32 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "172c5c91b0124f3a9cf694fc16662f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } health_report { overall_health: UNKNOWN } } }
I20260621 02:38:16.349581 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.344522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.0.97.3:42029)
0621 02:38:16.344688 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.344692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.344848 (+ 156us) server_negotiation.cc:407] Connection header received
0621 02:38:16.345113 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.345117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.345190 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.345394 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:16.345695 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.346343 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.347587 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.347986 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.348081 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.348540 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:16.349044 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:16.349150 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:16.349247 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 02:38:16.349371 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:38:16.357685 1811 raft_consensus.cc:1261] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Refusing update from remote peer 172c5c91b0124f3a9cf694fc16662f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:16.357738 1673 raft_consensus.cc:1261] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Refusing update from remote peer 172c5c91b0124f3a9cf694fc16662f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:16.358753 2025 consensus_queue.cc:1035] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:16.359145 2022 consensus_queue.cc:1035] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:16.375761 2028 mvcc.cc:205] Tried to move back new op lower bound from 7299110897071230976 to 7299110896691392512. Current Snapshot: MvccSnapshot[applied={T|T < 7299110897071230976}]
I20260621 02:38:16.405064 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:16.405342 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:16.408597 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646: Sending DeleteTablet for 3 replicas of tablet 75b2b814ca524796b7b3c43b0352f6e5
I20260621 02:38:16.409793 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b2b814ca524796b7b3c43b0352f6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58438
I20260621 02:38:16.410001 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b2b814ca524796b7b3c43b0352f6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39642
I20260621 02:38:16.411149 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b2b814ca524796b7b3c43b0352f6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37788
I20260621 02:38:16.412075 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:38:16.412748 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:16.420447 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 2148a1bc35a94f468c88253e8eed3f1a (DEFAULT_TABLE table=test-table [id=b904c61693b94b4fb4341161c7f3bde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:16.420807 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 2148a1bc35a94f468c88253e8eed3f1a (DEFAULT_TABLE table=test-table [id=b904c61693b94b4fb4341161c7f3bde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:16.421161 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2148a1bc35a94f468c88253e8eed3f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:16.421412 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2148a1bc35a94f468c88253e8eed3f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:16.426632 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 2148a1bc35a94f468c88253e8eed3f1a (DEFAULT_TABLE table=test-table [id=b904c61693b94b4fb4341161c7f3bde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:16.427186 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2148a1bc35a94f468c88253e8eed3f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:16.429029 2040 tablet_replica.cc:330] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: stopping tablet replica
I20260621 02:38:16.430142 2040 raft_consensus.cc:2229] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:16.431416 2015 tablet_bootstrap.cc:492] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: Bootstrap starting.
I20260621 02:38:16.432713 2040 raft_consensus.cc:2258] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:16.433799 2041 tablet_replica.cc:330] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: stopping tablet replica
I20260621 02:38:16.433988 2015 tablet_bootstrap.cc:654] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:16.434510 2039 tablet_replica.cc:330] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: stopping tablet replica
I20260621 02:38:16.437026 2040 ts_tablet_manager.cc:1905] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:16.437728 2039 raft_consensus.cc:2229] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:16.437597 2016 tablet_bootstrap.cc:492] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: Bootstrap starting.
I20260621 02:38:16.440294 2016 tablet_bootstrap.cc:654] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:16.442684 2041 raft_consensus.cc:2229] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:38:16.443648 2016 tablet_bootstrap.cc:492] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: No bootstrap required, opened a new log
I20260621 02:38:16.443892 2016 ts_tablet_manager.cc:1397] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 02:38:16.444293 2041 raft_consensus.cc:2258] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:16.438050 2017 tablet_bootstrap.cc:492] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: Bootstrap starting.
I20260621 02:38:16.444948 2016 raft_consensus.cc:348] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.445217 2016 raft_consensus.cc:374] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:16.445354 2016 raft_consensus.cc:729] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172c5c91b0124f3a9cf694fc16662f32, State: Initialized, Role: FOLLOWER
I20260621 02:38:16.445678 2016 consensus_queue.cc:260] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.446273 2017 tablet_bootstrap.cc:654] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:16.448871 2016 ts_tablet_manager.cc:1428] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260621 02:38:16.449831 2039 raft_consensus.cc:2258] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:16.451606 2017 tablet_bootstrap.cc:492] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: No bootstrap required, opened a new log
I20260621 02:38:16.451797 2017 ts_tablet_manager.cc:1397] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260621 02:38:16.452620 2017 raft_consensus.cc:348] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.452919 2017 raft_consensus.cc:374] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:16.453058 2017 raft_consensus.cc:729] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce5d4032aaa441ebcbc29ab40105b72, State: Initialized, Role: FOLLOWER
I20260621 02:38:16.454448 2017 consensus_queue.cc:260] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.454708 2039 ts_tablet_manager.cc:1905] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:16.456830 2040 ts_tablet_manager.cc:1918] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:16.456990 2040 log.cc:1196] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/75b2b814ca524796b7b3c43b0352f6e5
I20260621 02:38:16.444661 2015 tablet_bootstrap.cc:492] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: No bootstrap required, opened a new log
I20260621 02:38:16.457448 2015 ts_tablet_manager.cc:1397] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20260621 02:38:16.457401 2017 ts_tablet_manager.cc:1428] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260621 02:38:16.457581 2040 ts_tablet_manager.cc:1939] T 75b2b814ca524796b7b3c43b0352f6e5 P 1d6dce860a0e48b1bfac27366c25364d: Deleting consensus metadata
I20260621 02:38:16.459332 1498 catalog_manager.cc:4841] TS 1d6dce860a0e48b1bfac27366c25364d (127.0.97.1:36063): tablet 75b2b814ca524796b7b3c43b0352f6e5 (table test-table [id=ce108a409b424154a6abfcfb20e2169d]) successfully deleted
I20260621 02:38:16.459802 2041 ts_tablet_manager.cc:1905] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:16.459926 2015 raft_consensus.cc:348] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.460132 2015 raft_consensus.cc:374] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:16.460678 2015 raft_consensus.cc:729] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6dce860a0e48b1bfac27366c25364d, State: Initialized, Role: FOLLOWER
I20260621 02:38:16.460988 2015 consensus_queue.cc:260] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.461647 2015 ts_tablet_manager.cc:1428] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260621 02:38:16.466041 2039 ts_tablet_manager.cc:1918] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:16.466187 2039 log.cc:1196] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/75b2b814ca524796b7b3c43b0352f6e5
I20260621 02:38:16.466717 2039 ts_tablet_manager.cc:1939] T 75b2b814ca524796b7b3c43b0352f6e5 P cce5d4032aaa441ebcbc29ab40105b72: Deleting consensus metadata
I20260621 02:38:16.468443 1497 catalog_manager.cc:4841] TS cce5d4032aaa441ebcbc29ab40105b72 (127.0.97.2:33561): tablet 75b2b814ca524796b7b3c43b0352f6e5 (table test-table [id=ce108a409b424154a6abfcfb20e2169d]) successfully deleted
I20260621 02:38:16.469552 2041 ts_tablet_manager.cc:1918] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:16.469738 2041 log.cc:1196] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/75b2b814ca524796b7b3c43b0352f6e5
I20260621 02:38:16.470273 2041 ts_tablet_manager.cc:1939] T 75b2b814ca524796b7b3c43b0352f6e5 P 172c5c91b0124f3a9cf694fc16662f32: Deleting consensus metadata
I20260621 02:38:16.471773 1497 catalog_manager.cc:4841] TS 172c5c91b0124f3a9cf694fc16662f32 (127.0.97.3:42029): tablet 75b2b814ca524796b7b3c43b0352f6e5 (table test-table [id=ce108a409b424154a6abfcfb20e2169d]) successfully deleted
I20260621 02:38:16.585234 2022 raft_consensus.cc:482] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:16.585490 2022 raft_consensus.cc:504] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.586663 2022 leader_election.cc:288] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d6dce860a0e48b1bfac27366c25364d (127.0.97.1:36063), cce5d4032aaa441ebcbc29ab40105b72 (127.0.97.2:33561)
I20260621 02:38:16.586781 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2148a1bc35a94f468c88253e8eed3f1a" candidate_uuid: "172c5c91b0124f3a9cf694fc16662f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce5d4032aaa441ebcbc29ab40105b72" is_pre_election: true
I20260621 02:38:16.586781 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2148a1bc35a94f468c88253e8eed3f1a" candidate_uuid: "172c5c91b0124f3a9cf694fc16662f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dce860a0e48b1bfac27366c25364d" is_pre_election: true
I20260621 02:38:16.587034 1673 raft_consensus.cc:2454] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 172c5c91b0124f3a9cf694fc16662f32 in term 0.
I20260621 02:38:16.587050 1811 raft_consensus.cc:2454] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 172c5c91b0124f3a9cf694fc16662f32 in term 0.
I20260621 02:38:16.587476 1879 leader_election.cc:302] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 172c5c91b0124f3a9cf694fc16662f32, 1d6dce860a0e48b1bfac27366c25364d; no voters:
I20260621 02:38:16.587757 2022 raft_consensus.cc:2781] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:16.587877 2022 raft_consensus.cc:482] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:16.587970 2022 raft_consensus.cc:3037] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:16.591101 2022 raft_consensus.cc:504] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.591717 2022 leader_election.cc:288] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [CANDIDATE]: Term 1 election: Requested vote from peers 1d6dce860a0e48b1bfac27366c25364d (127.0.97.1:36063), cce5d4032aaa441ebcbc29ab40105b72 (127.0.97.2:33561)
I20260621 02:38:16.592267 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2148a1bc35a94f468c88253e8eed3f1a" candidate_uuid: "172c5c91b0124f3a9cf694fc16662f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dce860a0e48b1bfac27366c25364d"
I20260621 02:38:16.592267 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2148a1bc35a94f468c88253e8eed3f1a" candidate_uuid: "172c5c91b0124f3a9cf694fc16662f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce5d4032aaa441ebcbc29ab40105b72"
I20260621 02:38:16.592473 1811 raft_consensus.cc:3037] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:16.592482 1673 raft_consensus.cc:3037] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:16.595413 1811 raft_consensus.cc:2454] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 172c5c91b0124f3a9cf694fc16662f32 in term 1.
I20260621 02:38:16.595422 1673 raft_consensus.cc:2454] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 172c5c91b0124f3a9cf694fc16662f32 in term 1.
I20260621 02:38:16.595844 1879 leader_election.cc:302] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 172c5c91b0124f3a9cf694fc16662f32, 1d6dce860a0e48b1bfac27366c25364d; no voters:
I20260621 02:38:16.596114 2022 raft_consensus.cc:2781] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:16.596282 2022 raft_consensus.cc:686] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 1 LEADER]: Becoming Leader. State: Replica: 172c5c91b0124f3a9cf694fc16662f32, State: Running, Role: LEADER
I20260621 02:38:16.596590 2022 consensus_queue.cc:237] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.599259 1511 catalog_manager.cc:5495] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 172c5c91b0124f3a9cf694fc16662f32 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "172c5c91b0124f3a9cf694fc16662f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } health_report { overall_health: HEALTHY } } }
Jun 21 02:38:16 dist-test-slave-jkn7 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:16.662240 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.647677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.0.1:32896)
0621 02:38:16.648416 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:16.648443 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:16.648632 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:16.648996 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:16.649003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:16.649727 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:16.649909 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.649918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.651017 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.651029 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.652057 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.652069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.652748 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.652753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.652925 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.653719 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:16.653737 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:16.656257 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:16.659657 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:16.659670 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:16.659678 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:16.660085 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:16.660444 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:16.660449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:16.660453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:16.660589 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:16.661121 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:16.661129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:16.661327 (+ 198us) client_negotiation.cc:769] Sending connection context
0621 02:38:16.661517 (+ 190us) client_negotiation.cc:240] Negotiation successful
0621 02:38:16.661811 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":407,"threads_started":1}
I20260621 02:38:16.662304 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.647710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.0.97.62:37905)
0621 02:38:16.648722 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.648728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.648750 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:38:16.648799 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.648802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.648854 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.648965 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:16.650062 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.650864 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.652192 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.652591 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.652728 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.656476 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:16.656481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:16.656486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:16.656515 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:16.659516 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:16.660203 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:16.660211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:16.660215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:16.660330 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:16.660694 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:16.660702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:16.660705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:16.660978 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:16.661149 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:16.661609 (+ 460us) server_negotiation.cc:299] Negotiation successful
0621 02:38:16.661932 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":882,"thread_start_us":738,"threads_started":1}
I20260621 02:38:16.667884 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32896:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 02:38:16.668852 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:16.676203 1646 tablet_service.cc:1467] Processing CreateTablet for tablet e1ef08f8046340009d9a5a1ce49af610 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4adf3727bcf94fc7b98a5f1409059106]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:38:16.676738 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ef08f8046340009d9a5a1ce49af610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:16.676586 1786 tablet_service.cc:1467] Processing CreateTablet for tablet e1ef08f8046340009d9a5a1ce49af610 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4adf3727bcf94fc7b98a5f1409059106]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:38:16.677121 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ef08f8046340009d9a5a1ce49af610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:16.682124 1925 tablet_service.cc:1467] Processing CreateTablet for tablet e1ef08f8046340009d9a5a1ce49af610 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4adf3727bcf94fc7b98a5f1409059106]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:38:16.682662 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ef08f8046340009d9a5a1ce49af610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:16.682824 2015 tablet_bootstrap.cc:492] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: Bootstrap starting.
I20260621 02:38:16.684103 2015 tablet_bootstrap.cc:654] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:16.686726 2015 tablet_bootstrap.cc:492] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: No bootstrap required, opened a new log
I20260621 02:38:16.686882 2015 ts_tablet_manager.cc:1397] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260621 02:38:16.687567 2015 raft_consensus.cc:348] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.688198 2015 raft_consensus.cc:374] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:16.688290 2015 raft_consensus.cc:729] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6dce860a0e48b1bfac27366c25364d, State: Initialized, Role: FOLLOWER
I20260621 02:38:16.688529 2015 consensus_queue.cc:260] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.688990 2021 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.689378 2021 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:16.689782 2015 ts_tablet_manager.cc:1428] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 02:38:16.691344 2017 tablet_bootstrap.cc:492] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: Bootstrap starting.
I20260621 02:38:16.693058 2017 tablet_bootstrap.cc:654] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:16.693883 2016 tablet_bootstrap.cc:492] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: Bootstrap starting.
I20260621 02:38:16.694343 2021 raft_consensus.cc:482] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:16.694519 2021 raft_consensus.cc:504] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.696632 2016 tablet_bootstrap.cc:654] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:16.696691 2021 leader_election.cc:288] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cce5d4032aaa441ebcbc29ab40105b72 (127.0.97.2:33561), 172c5c91b0124f3a9cf694fc16662f32 (127.0.97.3:42029)
I20260621 02:38:16.705163 2016 tablet_bootstrap.cc:492] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: No bootstrap required, opened a new log
I20260621 02:38:16.705152 2017 tablet_bootstrap.cc:492] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: No bootstrap required, opened a new log
I20260621 02:38:16.705381 2016 ts_tablet_manager.cc:1397] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260621 02:38:16.705499 2017 ts_tablet_manager.cc:1397] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260621 02:38:16.706257 2016 raft_consensus.cc:348] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.706246 2017 raft_consensus.cc:348] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.706513 2016 raft_consensus.cc:374] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:16.706607 2016 raft_consensus.cc:729] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172c5c91b0124f3a9cf694fc16662f32, State: Initialized, Role: FOLLOWER
I20260621 02:38:16.706660 2017 raft_consensus.cc:374] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:16.706823 2017 raft_consensus.cc:729] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce5d4032aaa441ebcbc29ab40105b72, State: Initialized, Role: FOLLOWER
I20260621 02:38:16.706914 2016 consensus_queue.cc:260] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.707119 2017 consensus_queue.cc:260] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.707367 2022 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.707719 2022 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:16.707705 2023 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.708377 2017 ts_tablet_manager.cc:1428] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260621 02:38:16.709002 2016 ts_tablet_manager.cc:1428] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:16.708475 2023 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:16.719847 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.697580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39125 (local address 127.0.97.2:33561)
0621 02:38:16.697771 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.697777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.712691 (+ 14914us) server_negotiation.cc:407] Connection header received
0621 02:38:16.713675 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.713680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.713768 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.713856 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:16.714550 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.715442 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.717826 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.718727 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.718881 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.719013 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:16.719384 (+ 371us) server_negotiation.cc:299] Negotiation successful
0621 02:38:16.719522 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 02:38:16.719915 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.697172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33561 (local address 127.0.97.1:39125)
0621 02:38:16.712512 (+ 15340us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:16.712535 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:16.713469 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:16.713873 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:16.713878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:16.713910 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:16.714291 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.714304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.715683 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.715690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.717676 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.717687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.718945 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.718952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.719142 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.719200 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 02:38:16.719332 (+ 132us) client_negotiation.cc:240] Negotiation successful
0621 02:38:16.719479 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15171,"thread_start_us":15446,"threads_started":1}
I20260621 02:38:16.721066 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ef08f8046340009d9a5a1ce49af610" candidate_uuid: "1d6dce860a0e48b1bfac27366c25364d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce5d4032aaa441ebcbc29ab40105b72" is_pre_election: true
I20260621 02:38:16.721377 1811 raft_consensus.cc:2454] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d6dce860a0e48b1bfac27366c25364d in term 0.
I20260621 02:38:16.721899 1600 leader_election.cc:302] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6dce860a0e48b1bfac27366c25364d, cce5d4032aaa441ebcbc29ab40105b72; no voters:
I20260621 02:38:16.722360 2021 raft_consensus.cc:2781] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:16.722508 2021 raft_consensus.cc:482] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:16.722613 2021 raft_consensus.cc:3037] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:16.722723 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.713273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44015 (local address 127.0.97.3:42029)
0621 02:38:16.713479 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.713484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.713915 (+ 431us) server_negotiation.cc:407] Connection header received
0621 02:38:16.714211 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.714218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.714298 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.714417 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:16.715104 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.716606 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.719240 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.720854 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.721004 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.721098 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:16.721529 (+ 431us) server_negotiation.cc:299] Negotiation successful
0621 02:38:16.722431 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:38:16.723481 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.713119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42029 (local address 127.0.97.1:44015)
0621 02:38:16.713817 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:16.713838 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:16.714037 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:16.714490 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:16.714496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:16.714523 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:16.714888 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.714899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.716755 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.716761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.718977 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.718993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.721110 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.721117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.721340 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.721373 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:38:16.722964 (+ 1591us) client_negotiation.cc:240] Negotiation successful
0621 02:38:16.723119 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":437,"threads_started":1}
I20260621 02:38:16.724828 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ef08f8046340009d9a5a1ce49af610" candidate_uuid: "1d6dce860a0e48b1bfac27366c25364d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "172c5c91b0124f3a9cf694fc16662f32" is_pre_election: true
I20260621 02:38:16.725188 2021 raft_consensus.cc:504] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.725346 1946 raft_consensus.cc:2454] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d6dce860a0e48b1bfac27366c25364d in term 0.
I20260621 02:38:16.725893 2021 leader_election.cc:288] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [CANDIDATE]: Term 1 election: Requested vote from peers cce5d4032aaa441ebcbc29ab40105b72 (127.0.97.2:33561), 172c5c91b0124f3a9cf694fc16662f32 (127.0.97.3:42029)
I20260621 02:38:16.726441 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ef08f8046340009d9a5a1ce49af610" candidate_uuid: "1d6dce860a0e48b1bfac27366c25364d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "172c5c91b0124f3a9cf694fc16662f32"
I20260621 02:38:16.726631 1946 raft_consensus.cc:3037] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:16.726819 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ef08f8046340009d9a5a1ce49af610" candidate_uuid: "1d6dce860a0e48b1bfac27366c25364d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce5d4032aaa441ebcbc29ab40105b72"
I20260621 02:38:16.727048 1811 raft_consensus.cc:3037] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:16.729147 1946 raft_consensus.cc:2454] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d6dce860a0e48b1bfac27366c25364d in term 1.
I20260621 02:38:16.729543 1600 leader_election.cc:302] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 172c5c91b0124f3a9cf694fc16662f32, 1d6dce860a0e48b1bfac27366c25364d; no voters:
I20260621 02:38:16.729817 2021 raft_consensus.cc:2781] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:16.729981 1811 raft_consensus.cc:2454] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d6dce860a0e48b1bfac27366c25364d in term 1.
I20260621 02:38:16.730593 2021 raft_consensus.cc:686] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [term 1 LEADER]: Becoming Leader. State: Replica: 1d6dce860a0e48b1bfac27366c25364d, State: Running, Role: LEADER
I20260621 02:38:16.730983 2021 consensus_queue.cc:237] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } }
I20260621 02:38:16.733743 2058 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: TxnStatusTablet state changed. Reason: New leader 1d6dce860a0e48b1bfac27366c25364d. Latest consensus state: current_term: 1 leader_uuid: "1d6dce860a0e48b1bfac27366c25364d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.734016 2058 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:38:16.734688 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 02:38:16.734910 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 02:38:16.735338 1511 catalog_manager.cc:5495] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d6dce860a0e48b1bfac27366c25364d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1d6dce860a0e48b1bfac27366c25364d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } health_report { overall_health: UNKNOWN } } }
I20260621 02:38:16.758971 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.752187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36063 (local address 127.0.0.1:39648)
0621 02:38:16.752398 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:16.752419 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:16.752612 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:16.754081 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:16.754086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:16.754098 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:38:16.754278 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.754289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.755449 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.755455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.756478 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.756489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.757220 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.757226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.757440 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.757492 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:38:16.758381 (+ 889us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:38:16.758400 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 02:38:16.758494 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 02:38:16.758644 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 02:38:16.759311 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.752286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.0.97.1:36063)
0621 02:38:16.753720 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.753724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.753739 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:38:16.753811 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.753814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.753882 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.754063 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:16.754446 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.755229 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.756620 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.757069 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.757197 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.757679 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:16.758271 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:16.758422 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:16.758521 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 02:38:16.758595 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1335,"thread_start_us":1224,"threads_started":1}
I20260621 02:38:16.763231 1811 raft_consensus.cc:1261] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Refusing update from remote peer 1d6dce860a0e48b1bfac27366c25364d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:16.763464 1946 raft_consensus.cc:1261] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32 [term 1 FOLLOWER]: Refusing update from remote peer 1d6dce860a0e48b1bfac27366c25364d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:16.764318 2021 consensus_queue.cc:1035] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:16.764968 2058 consensus_queue.cc:1035] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d [LEADER]: Connected to new peer: Peer: permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:16.768618 2023 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: TxnStatusTablet state changed. Reason: New leader 1d6dce860a0e48b1bfac27366c25364d. Latest consensus state: current_term: 1 leader_uuid: "1d6dce860a0e48b1bfac27366c25364d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.768854 2023 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:16.769737 2021 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d6dce860a0e48b1bfac27366c25364d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.770010 2021 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:38:16.770331 2029 mvcc.cc:205] Tried to move back new op lower bound from 7299110898734641152 to 7299110898615599104. Current Snapshot: MvccSnapshot[applied={T|T < 7299110898734641152 or (T in {7299110898734641152})}]
I20260621 02:38:16.773630 2030 mvcc.cc:205] Tried to move back new op lower bound from 7299110898734641152 to 7299110898615599104. Current Snapshot: MvccSnapshot[applied={T|T < 7299110898734641152}]
I20260621 02:38:16.774036 2023 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d6dce860a0e48b1bfac27366c25364d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.774247 2023 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P cce5d4032aaa441ebcbc29ab40105b72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:16.782691 2058 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d6dce860a0e48b1bfac27366c25364d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.782951 2058 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P 1d6dce860a0e48b1bfac27366c25364d: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:38:16.785634 2025 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d6dce860a0e48b1bfac27366c25364d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.788987 2025 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:16.789510 2022 tablet_replica.cc:439] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: TxnStatusTablet state changed. Reason: New leader 1d6dce860a0e48b1bfac27366c25364d. Latest consensus state: current_term: 1 leader_uuid: "1d6dce860a0e48b1bfac27366c25364d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 } } peers { permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 } } peers { permanent_uuid: "172c5c91b0124f3a9cf694fc16662f32" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42029 } } }
I20260621 02:38:16.789817 2022 tablet_replica.cc:442] T e1ef08f8046340009d9a5a1ce49af610 P 172c5c91b0124f3a9cf694fc16662f32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:16.806298 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.797999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46825 (local address 127.0.97.1:36063)
0621 02:38:16.798163 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.798171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.798951 (+ 780us) server_negotiation.cc:407] Connection header received
0621 02:38:16.799166 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.799172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.799257 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.799361 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:16.800087 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.801833 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.804061 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.804707 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.804849 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.805435 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:16.805585 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:16.805686 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:16.805864 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 02:38:16.805958 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 02:38:16.806402 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.797998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36063 (local address 127.0.97.3:46825)
0621 02:38:16.798784 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:16.798805 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:16.798961 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:16.799408 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:16.799413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:16.799426 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:38:16.799854 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.799871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.801986 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.801992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.803894 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.803911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.804908 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.804915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.805124 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.805199 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:38:16.805705 (+ 506us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:38:16.805722 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 02:38:16.805815 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 02:38:16.805979 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":290,"threads_started":1}
I20260621 02:38:16.818398 1811 raft_consensus.cc:1261] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Refusing update from remote peer 172c5c91b0124f3a9cf694fc16662f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:16.818425 1673 raft_consensus.cc:1261] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Refusing update from remote peer 172c5c91b0124f3a9cf694fc16662f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:16.818992 2025 consensus_queue.cc:1035] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d6dce860a0e48b1bfac27366c25364d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:16.819468 2025 consensus_queue.cc:1035] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce5d4032aaa441ebcbc29ab40105b72" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:16.912328 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.892979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57373 (local address 127.0.97.3:42029)
0621 02:38:16.893137 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.893143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.905157 (+ 12014us) server_negotiation.cc:407] Connection header received
0621 02:38:16.905715 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.905722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.905805 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.905868 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:16.906640 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.907821 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.910006 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.910757 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.910851 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.911523 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:16.911650 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:16.911710 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:16.911935 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 02:38:16.912017 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:38:16.913113 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.892434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42029 (local address 127.0.97.1:57373)
0621 02:38:16.905040 (+ 12606us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:16.905063 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:16.905519 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:16.906060 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:16.906066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:16.906083 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:38:16.906430 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.906441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.907996 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.908002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.909856 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:16.909871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.911055 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.911061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:16.911242 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:16.911316 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:38:16.911794 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:38:16.911804 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:38:16.912583 (+ 779us) client_negotiation.cc:240] Negotiation successful
0621 02:38:16.912718 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12390,"thread_start_us":12739,"threads_started":1}
I20260621 02:38:16.970978 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:16.971211 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:16.974273 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1649a9bdb19a462ca660761628a2b646: Sending DeleteTablet for 3 replicas of tablet 2148a1bc35a94f468c88253e8eed3f1a
I20260621 02:38:16.974998 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 2148a1bc35a94f468c88253e8eed3f1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39642
I20260621 02:38:16.975217 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 2148a1bc35a94f468c88253e8eed3f1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58438
I20260621 02:38:16.975224 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 2148a1bc35a94f468c88253e8eed3f1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37788
I20260621 02:38:16.987638 2076 tablet_replica.cc:330] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: stopping tablet replica
I20260621 02:38:16.987913 2076 raft_consensus.cc:2229] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:16.988214 2076 raft_consensus.cc:2258] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:16.990581 2076 ts_tablet_manager.cc:1905] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:16.991309 2077 tablet_replica.cc:330] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: stopping tablet replica
I20260621 02:38:16.991722 2078 tablet_replica.cc:330] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: stopping tablet replica
I20260621 02:38:16.991966 2078 raft_consensus.cc:2229] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:16.992702 2077 raft_consensus.cc:2229] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:38:16.993028 2078 raft_consensus.cc:2258] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:16.993474 2077 raft_consensus.cc:2258] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:16.995028 2077 ts_tablet_manager.cc:1905] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:16.995433 2078 ts_tablet_manager.cc:1905] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:16.998425 2076 ts_tablet_manager.cc:1918] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:38:16.998586 2076 log.cc:1196] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/2148a1bc35a94f468c88253e8eed3f1a
I20260621 02:38:17.001605 2076 ts_tablet_manager.cc:1939] T 2148a1bc35a94f468c88253e8eed3f1a P 1d6dce860a0e48b1bfac27366c25364d: Deleting consensus metadata
I20260621 02:38:17.002655 1498 catalog_manager.cc:4841] TS 1d6dce860a0e48b1bfac27366c25364d (127.0.97.1:36063): tablet 2148a1bc35a94f468c88253e8eed3f1a (table test-table [id=b904c61693b94b4fb4341161c7f3bde8]) successfully deleted
Jun 21 02:38:17 dist-test-slave-jkn7 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:17.003599 2078 ts_tablet_manager.cc:1918] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:38:17.003731 2078 log.cc:1196] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/2148a1bc35a94f468c88253e8eed3f1a
I20260621 02:38:17.004176 2078 ts_tablet_manager.cc:1939] T 2148a1bc35a94f468c88253e8eed3f1a P cce5d4032aaa441ebcbc29ab40105b72: Deleting consensus metadata
I20260621 02:38:17.005144 2077 ts_tablet_manager.cc:1918] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:38:17.005249 1497 catalog_manager.cc:4841] TS cce5d4032aaa441ebcbc29ab40105b72 (127.0.97.2:33561): tablet 2148a1bc35a94f468c88253e8eed3f1a (table test-table [id=b904c61693b94b4fb4341161c7f3bde8]) successfully deleted
I20260621 02:38:17.005335 2077 log.cc:1196] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/2148a1bc35a94f468c88253e8eed3f1a
I20260621 02:38:17.005844 2077 ts_tablet_manager.cc:1939] T 2148a1bc35a94f468c88253e8eed3f1a P 172c5c91b0124f3a9cf694fc16662f32: Deleting consensus metadata
I20260621 02:38:17.006765 1497 catalog_manager.cc:4841] TS 172c5c91b0124f3a9cf694fc16662f32 (127.0.97.3:42029): tablet 2148a1bc35a94f468c88253e8eed3f1a (table test-table [id=b904c61693b94b4fb4341161c7f3bde8]) successfully deleted
I20260621 02:38:17.010964 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:16.990364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.0.97.1:36063)
0621 02:38:16.990537 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:16.990542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:16.992137 (+ 1595us) server_negotiation.cc:407] Connection header received
0621 02:38:16.992350 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:16.992356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:16.992463 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:16.992569 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:16.994398 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.995600 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.997503 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:16.998744 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:16.998868 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:17.003996 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:17.004014 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:17.004035 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:17.004078 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:17.008237 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.008900 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.008909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.008913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.009063 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.009476 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.009484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.009493 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.009817 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:17.010133 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:17.010426 (+ 293us) server_negotiation.cc:299] Negotiation successful
0621 02:38:17.010641 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260621 02:38:17.011686 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39662
I20260621 02:38:17.034649 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:17.020799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.0.97.62:37905)
0621 02:38:17.020994 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:17.021001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:17.021422 (+ 421us) server_negotiation.cc:407] Connection header received
0621 02:38:17.021577 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:17.021581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:17.021642 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:17.021718 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:17.023269 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:17.023928 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:17.025720 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:17.026282 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:17.026387 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:17.029180 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:17.029192 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:17.029195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:17.029226 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:17.032281 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.032855 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.032861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.032865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.032994 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.033319 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.033323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.033325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.033658 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:17.033901 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:17.034115 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 02:38:17.034314 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260621 02:38:17.035210 1512 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32898
I20260621 02:38:17.037900 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1583
I20260621 02:38:17.050379 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:17.038590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.0.97.1:36063)
0621 02:38:17.038785 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:17.038792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:17.039043 (+ 251us) server_negotiation.cc:407] Connection header received
0621 02:38:17.039222 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:17.039232 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:17.039306 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:17.039414 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:17.040425 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:17.041274 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:17.042460 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:17.042985 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:17.043115 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:17.045242 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:17.045248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:17.045256 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:17.045338 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:17.047891 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.048495 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.048501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.048507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.048632 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.048971 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.048977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.048980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.049272 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:17.049554 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:17.049839 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 02:38:17.050066 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 02:38:17.050884 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39666
W20260621 02:38:17.051391 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:38:17.051559 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 1583
I20260621 02:38:17.070833 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1722
Jun 21 02:38:17 dist-test-slave-jkn7 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:38:17.084792 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:17.071646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.0.97.2:33561)
0621 02:38:17.071792 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:17.071796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:17.071813 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:38:17.071863 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:17.071867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:17.071936 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:17.072030 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:17.073072 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:17.073895 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:17.075396 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:17.075774 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:17.075863 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:17.079863 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:17.079875 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:17.079890 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:17.079920 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:17.082648 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.083265 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.083272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.083275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.083412 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.083712 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.083715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.083717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.084017 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:17.084219 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:17.084411 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 02:38:17.084548 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260621 02:38:17.085522 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58440
W20260621 02:38:17.086091 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:38:17.086253 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 1722
I20260621 02:38:17.102748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1861
Jun 21 02:38:17 dist-test-slave-jkn7 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:38:17.116241 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:17.103415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.0.97.3:42029)
0621 02:38:17.103574 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:17.103578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:17.103655 (+ 77us) server_negotiation.cc:407] Connection header received
0621 02:38:17.103787 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:17.103791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:17.103858 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:17.103938 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:17.105063 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:17.105804 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:17.106941 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:17.107323 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:17.107411 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:17.111316 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:17.111329 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:17.111345 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:17.111373 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:17.114125 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.114684 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.114691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.114693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.114809 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:17.115124 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:17.115128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:17.115130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:17.115457 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:17.115698 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:17.115863 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 02:38:17.116017 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260621 02:38:17.116983 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806
W20260621 02:38:17.117717 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:38:17.117869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 1861
I20260621 02:38:17.138514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 1480
I20260621 02:38:17.139811 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:17.250154 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 1480
2026-06-21T02:38:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4671 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:17 dist-test-slave-jkn7 krb5kdc[2087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:17 dist-test-slave-jkn7 krb5kdc[2087](info): set up 2 sockets
Jun 21 02:38:17 dist-test-slave-jkn7 krb5kdc[2087](info): commencing operation
krb5kdc: starting...
W20260621 02:38:18.333446 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:18 dist-test-slave-jkn7 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:18.506539 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42405
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:18.817171 2100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:18.817541 2100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:18.817695 2100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:18.824908 2100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:18.825057 2100 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 02:38:18.825160 2100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:18.825273 2100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:18.825389 2100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:18.825477 2100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:18.835101 2100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42405
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:18.836349 2100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:18.837725 2100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:18.849197 2105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:18.849305 2106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:18.849190 2108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:18.850107 2100 server_base.cc:1034] running on GCE node
I20260621 02:38:18.850883 2100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:18.852192 2100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:18.853427 2100 hybrid_clock.cc:648] HybridClock initialized: now 1782009498853365 us; error 39 us; skew 500 ppm
I20260621 02:38:18.853777 2100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:18 dist-test-slave-jkn7 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:18.856853 2100 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:18.858242 2100 webserver.cc:460] Webserver started at http://127.0.97.62:33435/ using document root <none> and password file <none>
I20260621 02:38:18.858911 2100 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:18.859035 2100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:18.859349 2100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:18.862257 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "db53966ada334540a3de4ef7ba37c237"
format_stamp: "Formatted at 2026-06-21 02:38:18 on dist-test-slave-jkn7"
server_key: "9016eaaadee2b60ab90ecc977811c2a0"
server_key_iv: "f602289ea15341b20db726237897f060"
server_key_version: "encryptionkey@0"
I20260621 02:38:18.863005 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "db53966ada334540a3de4ef7ba37c237"
format_stamp: "Formatted at 2026-06-21 02:38:18 on dist-test-slave-jkn7"
server_key: "9016eaaadee2b60ab90ecc977811c2a0"
server_key_iv: "f602289ea15341b20db726237897f060"
server_key_version: "encryptionkey@0"
I20260621 02:38:18.867820 2100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:38:18.871178 2115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:18.872431 2100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:18.872593 2100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "db53966ada334540a3de4ef7ba37c237"
format_stamp: "Formatted at 2026-06-21 02:38:18 on dist-test-slave-jkn7"
server_key: "9016eaaadee2b60ab90ecc977811c2a0"
server_key_iv: "f602289ea15341b20db726237897f060"
server_key_version: "encryptionkey@0"
I20260621 02:38:18.872776 2100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:18.902678 2100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:18.906814 2100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:18.907094 2100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:18.928399 2100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42405
I20260621 02:38:18.928417 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42405 every 8 connection(s)
I20260621 02:38:18.929720 2100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:18.931975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 2100
I20260621 02:38:18.932191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:18.932787 388 external_mini_cluster.cc:1428] Setting key ba3cc080f4c89c209324e6bd523be88a
I20260621 02:38:18.933617 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:38:18 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:18.947137 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237: Bootstrap starting.
I20260621 02:38:18.951314 2177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:18.952557 2177 log.cc:824] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:18.955380 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237: No bootstrap required, opened a new log
I20260621 02:38:18.956002 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:18.934722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.0.97.62:42405)
0621 02:38:18.935453 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:18.935467 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:18.935518 (+ 51us) server_negotiation.cc:407] Connection header received
0621 02:38:18.937194 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:18.937221 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:18.937745 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:18.938170 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:18.939350 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:18.941424 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:18.942605 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:18.943591 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:18.943783 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:18.947823 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:18.947833 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:18.947852 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:18.947907 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:18.951873 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:18.952568 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:18.952579 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:18.952588 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:18.952785 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:18.953120 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:18.953126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:18.953129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:18.953745 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:18.954039 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:18.954502 (+ 463us) server_negotiation.cc:299] Negotiation successful
0621 02:38:18.954944 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":319,"threads_started":1}
I20260621 02:38:18.962752 2177 raft_consensus.cc:348] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db53966ada334540a3de4ef7ba37c237" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } }
I20260621 02:38:18.963120 2177 raft_consensus.cc:374] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:18.963228 2177 raft_consensus.cc:729] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db53966ada334540a3de4ef7ba37c237, State: Initialized, Role: FOLLOWER
I20260621 02:38:18.963748 2177 consensus_queue.cc:260] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db53966ada334540a3de4ef7ba37c237" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } }
I20260621 02:38:18.963984 2177 raft_consensus.cc:388] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:18.964124 2177 raft_consensus.cc:482] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:18.964299 2177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:18.966419 2177 raft_consensus.cc:504] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db53966ada334540a3de4ef7ba37c237" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } }
I20260621 02:38:18.966907 2177 leader_election.cc:302] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db53966ada334540a3de4ef7ba37c237; no voters:
I20260621 02:38:18.967429 2177 leader_election.cc:288] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:18.967551 2182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:18.968333 2182 raft_consensus.cc:686] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [term 1 LEADER]: Becoming Leader. State: Replica: db53966ada334540a3de4ef7ba37c237, State: Running, Role: LEADER
I20260621 02:38:18.968866 2182 consensus_queue.cc:237] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db53966ada334540a3de4ef7ba37c237" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } }
I20260621 02:38:18.969617 2177 sys_catalog.cc:564] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:18.971493 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db53966ada334540a3de4ef7ba37c237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db53966ada334540a3de4ef7ba37c237" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } } }
I20260621 02:38:18.971868 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:18.974632 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db53966ada334540a3de4ef7ba37c237. Latest consensus state: current_term: 1 leader_uuid: "db53966ada334540a3de4ef7ba37c237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db53966ada334540a3de4ef7ba37c237" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } } }
I20260621 02:38:18.974912 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:18.975581 2189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:18.980979 2189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:18.989624 2189 catalog_manager.cc:1347] Generated new cluster ID: b1d4d6e341d144d3a968ff45aed5d120
I20260621 02:38:18.989756 2189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:19.009680 2189 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:19.011623 2189 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:19.025336 2189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237: Generated new TSK 0
I20260621 02:38:19.026266 2189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:38:19.074679 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42405
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:19.392673 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:19.392937 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:19.393035 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:19.400104 2203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:19.400209 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:19.400291 2203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 02:38:19.400343 2203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:38:19.400406 2203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:19.410297 2203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:19.411569 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:19.412829 2203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:19.425544 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:19.425583 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:19.425626 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:19.426517 2203 server_base.cc:1034] running on GCE node
I20260621 02:38:19.427110 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:19.427992 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:19.429214 2203 hybrid_clock.cc:648] HybridClock initialized: now 1782009499429156 us; error 50 us; skew 500 ppm
I20260621 02:38:19.429533 2203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:19 dist-test-slave-jkn7 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:19.432498 2203 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:38:19.434012 2203 webserver.cc:460] Webserver started at http://127.0.97.1:46071/ using document root <none> and password file <none>
I20260621 02:38:19.434675 2203 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:19.434818 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:19.435125 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:19.437950 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "d2501fe77d7e49199da3ae1f5ba9cc88"
format_stamp: "Formatted at 2026-06-21 02:38:19 on dist-test-slave-jkn7"
server_key: "2e4bd65155f5611eea4940892f14ce17"
server_key_iv: "7a2d7fe2c8da25afae08877dbccd8496"
server_key_version: "encryptionkey@0"
I20260621 02:38:19.438732 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d2501fe77d7e49199da3ae1f5ba9cc88"
format_stamp: "Formatted at 2026-06-21 02:38:19 on dist-test-slave-jkn7"
server_key: "2e4bd65155f5611eea4940892f14ce17"
server_key_iv: "7a2d7fe2c8da25afae08877dbccd8496"
server_key_version: "encryptionkey@0"
I20260621 02:38:19.443601 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 02:38:19.446878 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:19.448117 2203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:38:19.448288 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "d2501fe77d7e49199da3ae1f5ba9cc88"
format_stamp: "Formatted at 2026-06-21 02:38:19 on dist-test-slave-jkn7"
server_key: "2e4bd65155f5611eea4940892f14ce17"
server_key_iv: "7a2d7fe2c8da25afae08877dbccd8496"
server_key_version: "encryptionkey@0"
I20260621 02:38:19.448496 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:19.471890 2203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:19.476459 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:19.476753 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:19.479291 2203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:19.479380 2203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:19.479543 2203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:19.479637 2203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:38:19 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:19.505752 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:19.480586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42405 (local address 127.0.97.1:47765)
0621 02:38:19.481527 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:19.481838 (+ 311us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:19.483830 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:19.484668 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:19.484679 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:19.485429 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:19.486201 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:19.486219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:19.487963 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:19.487969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:19.490238 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:19.490250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:19.491591 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:19.491603 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:19.491960 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:19.492869 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:19.492909 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:19.496986 (+ 4077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:19.500929 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:19.500938 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:19.500970 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:19.501563 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:19.501936 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:19.501941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:19.501945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:19.502190 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:19.502848 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:19.502861 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:19.503248 (+ 387us) client_negotiation.cc:769] Sending connection context
0621 02:38:19.503629 (+ 381us) client_negotiation.cc:240] Negotiation successful
0621 02:38:19.504194 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":423,"threads_started":1}
I20260621 02:38:19.506386 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:19.479996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47765 (local address 127.0.97.62:42405)
0621 02:38:19.482242 (+ 2246us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:19.482248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:19.482270 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:38:19.484213 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:19.484222 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:19.484333 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:19.484487 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:19.486481 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:19.487765 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:19.490357 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:19.491113 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:19.491269 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:19.497324 (+ 6055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:19.497333 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:19.497338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:19.497384 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:19.500784 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:19.501695 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:19.501701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:19.501705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:19.501820 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:19.502315 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:19.502320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:19.502324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:19.502593 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:19.502764 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:19.505495 (+ 2731us) server_negotiation.cc:299] Negotiation successful
0621 02:38:19.505793 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2124,"thread_start_us":2020,"threads_started":1}
I20260621 02:38:19.522294 2203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34105
I20260621 02:38:19.522336 2338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34105 every 8 connection(s)
I20260621 02:38:19.523655 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:19.525853 2339 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42405
I20260621 02:38:19.526081 2339 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:19.526865 2339 heartbeater.cc:511] Master 127.0.97.62:42405 requested a full tablet report, sending...
I20260621 02:38:19.528491 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:19.513492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42405 (local address 127.0.97.1:33301)
0621 02:38:19.514392 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:19.514423 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:19.514632 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:19.515246 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:19.515256 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:19.516112 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:19.516620 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:19.516637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:19.517942 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:19.517951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:19.519953 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:19.519970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:19.520739 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:19.520747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:19.520901 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:19.521649 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:19.521667 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:19.522528 (+ 861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:19.526069 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:19.526073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:19.526076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:19.526490 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:19.526882 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:19.526885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:19.526887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:19.527036 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:19.527613 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:19.527618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:19.527816 (+ 198us) client_negotiation.cc:769] Sending connection context
0621 02:38:19.527943 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 02:38:19.528209 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":462,"threads_started":1}
I20260621 02:38:19.529059 2132 ts_manager.cc:194] Registered new tserver with Master: d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1:34105)
I20260621 02:38:19.529363 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:19.513444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33301 (local address 127.0.97.62:42405)
0621 02:38:19.513657 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:19.513665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:19.514508 (+ 843us) server_negotiation.cc:407] Connection header received
0621 02:38:19.514809 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:19.514815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:19.514913 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:19.514997 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:19.516820 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:19.517792 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:19.520126 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:19.520526 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:19.520668 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:19.522702 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:19.522708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:19.522713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:19.522751 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:19.525953 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:19.526612 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:19.526618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:19.526621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:19.526766 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:19.527127 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:19.527131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:19.527134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:19.527455 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:19.528740 (+ 1285us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:19.528861 (+ 121us) server_negotiation.cc:299] Negotiation successful
0621 02:38:19.529007 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:38:19.531011 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47765
I20260621 02:38:19.531903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 2203
I20260621 02:38:19.532083 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:19.532708 388 external_mini_cluster.cc:1428] Setting key 0461fc7b7fdf4b34c0636aa3053ee43d
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:38:19.570398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42405
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:19.888201 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:19.888474 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:19.888664 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:19.895974 2343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:19.896078 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:19.896216 2343 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 02:38:19.896276 2343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:38:19.896384 2343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:19.905972 2343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:19.907310 2343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:19.908879 2343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:19.922022 2351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:19.922046 2349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:19.922077 2348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:19.922880 2343 server_base.cc:1034] running on GCE node
I20260621 02:38:19.923491 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:19.924432 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:19.925632 2343 hybrid_clock.cc:648] HybridClock initialized: now 1782009499925583 us; error 33 us; skew 500 ppm
I20260621 02:38:19.925917 2343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:19 dist-test-slave-jkn7 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:19.929229 2343 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:38:19.943414 2343 webserver.cc:460] Webserver started at http://127.0.97.2:38701/ using document root <none> and password file <none>
I20260621 02:38:19.944175 2343 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:19.944329 2343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:19.944648 2343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:19.947817 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "f68fddd4fd494c32ad9846e399193e15"
format_stamp: "Formatted at 2026-06-21 02:38:19 on dist-test-slave-jkn7"
server_key: "aec37d98c73fc9ba0f0a2223bfad2ec1"
server_key_iv: "487f2a770346509f3939bef3b9e25c1c"
server_key_version: "encryptionkey@0"
I20260621 02:38:19.948599 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f68fddd4fd494c32ad9846e399193e15"
format_stamp: "Formatted at 2026-06-21 02:38:19 on dist-test-slave-jkn7"
server_key: "aec37d98c73fc9ba0f0a2223bfad2ec1"
server_key_iv: "487f2a770346509f3939bef3b9e25c1c"
server_key_version: "encryptionkey@0"
I20260621 02:38:19.953788 2343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260621 02:38:19.957142 2358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:19.958490 2343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:19.958666 2343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "f68fddd4fd494c32ad9846e399193e15"
format_stamp: "Formatted at 2026-06-21 02:38:19 on dist-test-slave-jkn7"
server_key: "aec37d98c73fc9ba0f0a2223bfad2ec1"
server_key_iv: "487f2a770346509f3939bef3b9e25c1c"
server_key_version: "encryptionkey@0"
I20260621 02:38:19.958854 2343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:20.002590 2343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:20.007014 2343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:20.007332 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:20.010339 2343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:20.010465 2343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:20.010573 2343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:20.010658 2343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:38:20 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:20.032145 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.011040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52597 (local address 127.0.97.62:42405)
0621 02:38:20.011217 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.011221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.012888 (+ 1667us) server_negotiation.cc:407] Connection header received
0621 02:38:20.015129 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.015135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.015222 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.015345 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:20.016907 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.017767 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.020381 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.020789 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.020921 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.025501 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:20.025510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:20.025516 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:20.025559 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:20.028254 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.028991 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.028999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.029006 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.029131 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.029752 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.029756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.029758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.030129 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:20.030284 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.031329 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.031633 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:38:20.033087 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.011858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42405 (local address 127.0.97.2:52597)
0621 02:38:20.012729 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.012773 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.014827 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.015616 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.015627 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.016138 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:20.016733 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.016751 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.017913 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.017921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.020219 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.020233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.021037 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.021046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.021417 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.022186 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:20.022236 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:20.025162 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:20.028402 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.028410 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.028426 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.028874 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.029248 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.029253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.029256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.029588 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.030345 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:20.030354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:20.030842 (+ 488us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.031156 (+ 314us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.031651 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":408,"threads_started":1}
I20260621 02:38:20.056600 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.038704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42405 (local address 127.0.97.2:57277)
0621 02:38:20.039441 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.039466 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.039639 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.039948 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.039955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.040657 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:20.041106 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.041121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.042419 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.042431 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.044647 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.044668 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.045458 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.045467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.046284 (+ 817us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.046963 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:20.046987 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:20.048392 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:20.052841 (+ 4449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.052847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.052854 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.053545 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.054052 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.054059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.054062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.054243 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.055053 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:20.055060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:20.055315 (+ 255us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.055503 (+ 188us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.055927 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":378,"threads_started":1}
I20260621 02:38:20.057767 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.038723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57277 (local address 127.0.97.62:42405)
0621 02:38:20.038926 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.038935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.039562 (+ 627us) server_negotiation.cc:407] Connection header received
0621 02:38:20.039752 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.039755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.039809 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.039884 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:20.041319 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.042268 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.044796 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.045203 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.045375 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.048650 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:20.048660 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:20.048668 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:20.048716 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:20.052672 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.053742 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.053750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.053756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.053896 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.054418 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.054425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.054431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.054895 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:20.056910 (+ 2015us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.057058 (+ 148us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.057357 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:38:20.069408 2343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37587
I20260621 02:38:20.069448 2477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37587 every 8 connection(s)
I20260621 02:38:20.070595 2343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:20.072549 2478 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42405
I20260621 02:38:20.072764 2478 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:20.073451 2478 heartbeater.cc:511] Master 127.0.97.62:42405 requested a full tablet report, sending...
I20260621 02:38:20.074828 2132 ts_manager.cc:194] Registered new tserver with Master: f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587)
I20260621 02:38:20.075644 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52597
I20260621 02:38:20.078676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 2343
I20260621 02:38:20.078804 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:20.079160 388 external_mini_cluster.cc:1428] Setting key 84e957b2ed15e39025200809958704eb
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:38:20.115721 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42405
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:20.428966 2482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:20.429276 2482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:20.429471 2482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:20.436365 2482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:20.436465 2482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:20.436544 2482 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 02:38:20.436595 2482 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:38:20.436653 2482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:20.446014 2482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:20.447232 2482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:20.448470 2482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:20.461544 2488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:20.461546 2490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:20.461607 2487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:20.462374 2482 server_base.cc:1034] running on GCE node
I20260621 02:38:20.463004 2482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:20.463899 2482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:20.465091 2482 hybrid_clock.cc:648] HybridClock initialized: now 1782009500465038 us; error 47 us; skew 500 ppm
I20260621 02:38:20.465456 2482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:20 dist-test-slave-jkn7 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:20.471350 2482 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:38:20.472674 2482 webserver.cc:460] Webserver started at http://127.0.97.3:39097/ using document root <none> and password file <none>
I20260621 02:38:20.473364 2482 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:20.473465 2482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:20.473722 2482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:20.476886 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "5b8b00ef016942a181ecac06da99bb39"
format_stamp: "Formatted at 2026-06-21 02:38:20 on dist-test-slave-jkn7"
server_key: "305201e9eb666a37dbd4b764a5e88c55"
server_key_iv: "13e8e06fb0f6b3f12afbab93c19d3e7b"
server_key_version: "encryptionkey@0"
I20260621 02:38:20.477676 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5b8b00ef016942a181ecac06da99bb39"
format_stamp: "Formatted at 2026-06-21 02:38:20 on dist-test-slave-jkn7"
server_key: "305201e9eb666a37dbd4b764a5e88c55"
server_key_iv: "13e8e06fb0f6b3f12afbab93c19d3e7b"
server_key_version: "encryptionkey@0"
I20260621 02:38:20.482601 2482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 02:38:20.486016 2497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:20.487413 2482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:38:20.487571 2482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "5b8b00ef016942a181ecac06da99bb39"
format_stamp: "Formatted at 2026-06-21 02:38:20 on dist-test-slave-jkn7"
server_key: "305201e9eb666a37dbd4b764a5e88c55"
server_key_iv: "13e8e06fb0f6b3f12afbab93c19d3e7b"
server_key_version: "encryptionkey@0"
I20260621 02:38:20.487759 2482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:20.506230 2482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:20.510766 2482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:20.511062 2482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:20.513789 2482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:20.513881 2482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:20.514016 2482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:20.514075 2482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:38:20 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:20.535490 2339 heartbeater.cc:503] Master 127.0.97.62:42405 was elected leader, sending a full tablet report...
I20260621 02:38:20.539008 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.514706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59997 (local address 127.0.97.62:42405)
0621 02:38:20.514886 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.514892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.516491 (+ 1599us) server_negotiation.cc:407] Connection header received
0621 02:38:20.518753 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.518765 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.518859 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.518973 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:20.521354 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.522610 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.525721 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.526177 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.526314 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.531440 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:20.531446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:20.531450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:20.531484 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:20.534948 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.536006 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.536015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.536022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.536242 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.536790 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.536797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.536803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.537312 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:20.537494 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.538180 (+ 686us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.538482 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:38:20.539525 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.515418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42405 (local address 127.0.97.3:59997)
0621 02:38:20.516310 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.516353 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.518453 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.519220 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.519230 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.519963 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:20.520977 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.521003 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.522837 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.522845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.525527 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.525545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.526421 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.526429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.526772 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.527686 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:20.527722 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:20.531175 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:20.535174 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.535183 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.535199 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.535832 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.536382 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.536386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.536390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.536658 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.537437 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:20.537443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:20.537773 (+ 330us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.537989 (+ 216us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.538362 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":435,"threads_started":1}
I20260621 02:38:20.559748 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.545274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42405 (local address 127.0.97.3:59621)
0621 02:38:20.546180 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.546196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.546348 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.546678 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.546683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.547331 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:20.547866 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.547884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.549210 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.549215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.551073 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.551084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.551903 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.551909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.552081 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.552864 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:20.552880 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:20.553750 (+ 870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:20.556823 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.556857 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.556863 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.557477 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.557856 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.557864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.557868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.557999 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.558492 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:20.558497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:20.558738 (+ 241us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.558913 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.559250 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":578,"threads_started":1}
I20260621 02:38:20.560551 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.545187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59621 (local address 127.0.97.62:42405)
0621 02:38:20.545413 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.545421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.546278 (+ 857us) server_negotiation.cc:407] Connection header received
0621 02:38:20.546481 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.546487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.546557 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.546654 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:20.548061 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.549025 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.551212 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.551723 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.551864 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.553973 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:20.553982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:20.553989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:20.554023 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:20.556697 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.557621 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.557630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.557632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.557746 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.558123 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.558128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.558131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.558371 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:20.558531 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.560128 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.560327 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:38:20.570495 2482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35265
I20260621 02:38:20.570528 2616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35265 every 8 connection(s)
I20260621 02:38:20.571710 2482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:20.572227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 2482
I20260621 02:38:20.572408 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:20.572997 388 external_mini_cluster.cc:1428] Setting key 1a782bc3c14c401df1fe9d4e8fc2a67f
I20260621 02:38:20.573863 2617 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42405
I20260621 02:38:20.574115 2617 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:20.574833 2617 heartbeater.cc:511] Master 127.0.97.62:42405 requested a full tablet report, sending...
I20260621 02:38:20.576349 2132 ts_manager.cc:194] Registered new tserver with Master: 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:20.577128 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59997
I20260621 02:38:20.588318 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:38:20 dist-test-slave-jkn7 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:38:20 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:20.632259 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.619037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.0.97.62:42405)
0621 02:38:20.619275 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.619282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.619494 (+ 212us) server_negotiation.cc:407] Connection header received
0621 02:38:20.619708 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.619716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.619783 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.619889 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:20.621236 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.622021 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.623240 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.623666 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.623798 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.627614 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:20.627620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:20.627623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:20.627652 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:20.629998 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.630562 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.630567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.630569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.630682 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.631022 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.631025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.631027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.631261 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:20.631439 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.631678 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.631830 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:38:20.636305 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:38:20.644243 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:20.668232 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.656125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37587 (local address 127.0.0.1:35442)
0621 02:38:20.657142 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.657196 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.657416 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.658483 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.658490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.658530 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:20.659054 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.659070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.661166 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.661171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.664342 (+ 3171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.664354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.666959 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.666968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.667275 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.667347 (+ 72us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.667505 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.667770 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":576,"threads_started":1}
I20260621 02:38:20.668967 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.656406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.0.97.2:37587)
0621 02:38:20.656981 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.656991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.657609 (+ 618us) server_negotiation.cc:407] Connection header received
0621 02:38:20.657727 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.657735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.658008 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.658197 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:20.659301 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.661022 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.664520 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.666381 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.666544 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.666807 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.668468 (+ 1661us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.668657 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":1268,"threads_started":1}
I20260621 02:38:20.673342 2413 tablet_service.cc:1467] Processing CreateTablet for tablet e74fceaa0c3246a39de22e208e627be2 (DEFAULT_TABLE table=test-table [id=c3fdd900156047afa335720d37be0147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:20.674831 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74fceaa0c3246a39de22e208e627be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:20.677918 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.656923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34105 (local address 127.0.0.1:49394)
0621 02:38:20.657559 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.657575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.657685 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.671290 (+ 13605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.671295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.671323 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:20.671660 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.671675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.673684 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.673756 (+ 72us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.675573 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.675593 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.677432 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.677436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.677544 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.677567 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.677616 (+ 49us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.677696 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":915,"threads_started":1}
I20260621 02:38:20.678763 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.657526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.97.1:34105)
0621 02:38:20.670735 (+ 13209us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.670742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.670765 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:38:20.670840 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.670845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.671038 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.671316 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:20.671852 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.673549 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.675783 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.677212 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.678158 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.678330 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.678379 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.678490 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13061,"thread_start_us":12929,"threads_started":1}
I20260621 02:38:20.680401 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.657557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35265 (local address 127.0.0.1:47430)
0621 02:38:20.668327 (+ 10770us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.668347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.668463 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.672571 (+ 4108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.672575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.672600 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:20.672934 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.672946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.675038 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.675044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.677175 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.677189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.679657 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.679663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.679833 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.679873 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.679968 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.680098 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10593,"thread_start_us":10975,"threads_started":1}
I20260621 02:38:20.680485 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.658176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.0.97.3:35265)
0621 02:38:20.672006 (+ 13830us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.672014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.672047 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:38:20.672143 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.672150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.672369 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.672559 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:20.673159 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.674786 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.677376 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.679450 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.679586 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.679788 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.680022 (+ 234us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.680168 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13646,"thread_start_us":13466,"threads_started":1}
I20260621 02:38:20.681952 2269 tablet_service.cc:1467] Processing CreateTablet for tablet e74fceaa0c3246a39de22e208e627be2 (DEFAULT_TABLE table=test-table [id=c3fdd900156047afa335720d37be0147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:20.683255 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74fceaa0c3246a39de22e208e627be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:20.683888 2547 tablet_service.cc:1467] Processing CreateTablet for tablet e74fceaa0c3246a39de22e208e627be2 (DEFAULT_TABLE table=test-table [id=c3fdd900156047afa335720d37be0147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:20.685166 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74fceaa0c3246a39de22e208e627be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:20.687409 2636 tablet_bootstrap.cc:492] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: Bootstrap starting.
I20260621 02:38:20.690265 2636 tablet_bootstrap.cc:654] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:20.691366 2636 log.cc:824] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:20.695597 2636 tablet_bootstrap.cc:492] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: No bootstrap required, opened a new log
I20260621 02:38:20.695854 2636 ts_tablet_manager.cc:1397] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260621 02:38:20.696951 2638 tablet_bootstrap.cc:492] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: Bootstrap starting.
I20260621 02:38:20.698786 2639 tablet_bootstrap.cc:492] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: Bootstrap starting.
I20260621 02:38:20.700484 2638 tablet_bootstrap.cc:654] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:20.701448 2639 tablet_bootstrap.cc:654] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:20.701709 2638 log.cc:824] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:20.702509 2639 log.cc:824] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:20.704563 2638 tablet_bootstrap.cc:492] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: No bootstrap required, opened a new log
I20260621 02:38:20.704850 2639 tablet_bootstrap.cc:492] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: No bootstrap required, opened a new log
I20260621 02:38:20.705067 2638 ts_tablet_manager.cc:1397] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260621 02:38:20.705374 2639 ts_tablet_manager.cc:1397] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260621 02:38:20.705933 2636 raft_consensus.cc:348] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } }
I20260621 02:38:20.706355 2636 raft_consensus.cc:374] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:20.706460 2636 raft_consensus.cc:729] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68fddd4fd494c32ad9846e399193e15, State: Initialized, Role: FOLLOWER
I20260621 02:38:20.707060 2636 consensus_queue.cc:260] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } }
I20260621 02:38:20.708113 2478 heartbeater.cc:503] Master 127.0.97.62:42405 was elected leader, sending a full tablet report...
I20260621 02:38:20.708675 2636 ts_tablet_manager.cc:1428] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260621 02:38:20.713928 2639 raft_consensus.cc:348] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } }
I20260621 02:38:20.714327 2639 raft_consensus.cc:374] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:20.714421 2639 raft_consensus.cc:729] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8b00ef016942a181ecac06da99bb39, State: Initialized, Role: FOLLOWER
I20260621 02:38:20.714687 2638 raft_consensus.cc:348] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } }
I20260621 02:38:20.714984 2638 raft_consensus.cc:374] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:20.715086 2638 raft_consensus.cc:729] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2501fe77d7e49199da3ae1f5ba9cc88, State: Initialized, Role: FOLLOWER
I20260621 02:38:20.715010 2639 consensus_queue.cc:260] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } }
I20260621 02:38:20.715700 2638 consensus_queue.cc:260] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } }
I20260621 02:38:20.716120 2617 heartbeater.cc:503] Master 127.0.97.62:42405 was elected leader, sending a full tablet report...
I20260621 02:38:20.716674 2639 ts_tablet_manager.cc:1428] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260621 02:38:20.718571 2638 ts_tablet_manager.cc:1428] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260621 02:38:20.730945 2644 raft_consensus.cc:482] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:20.731173 2644 raft_consensus.cc:504] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } }
I20260621 02:38:20.732625 2644 leader_election.cc:288] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:20.739202 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.733009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37347 (local address 127.0.97.2:37587)
0621 02:38:20.733204 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.733209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.733744 (+ 535us) server_negotiation.cc:407] Connection header received
0621 02:38:20.734047 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.734057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.734150 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.734224 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:20.735002 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.735785 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.737566 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.738303 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.738457 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.738559 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.738819 (+ 260us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.738935 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:38:20.739305 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.732943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37587 (local address 127.0.97.1:37347)
0621 02:38:20.733653 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.733674 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.733852 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.734437 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.734442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.734478 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:20.734871 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.734881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.735921 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.735929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.737432 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.737443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.738507 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.738513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.738671 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.738693 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.738796 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.738925 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":1,"thread_start_us":430,"threads_started":1}
I20260621 02:38:20.739627 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.733953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40853 (local address 127.0.97.3:35265)
0621 02:38:20.734165 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.734171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.734732 (+ 561us) server_negotiation.cc:407] Connection header received
0621 02:38:20.734921 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.734926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.734993 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.735085 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:20.735607 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.736332 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.737984 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.738722 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.738863 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.738967 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.739239 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.739368 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:38:20.739954 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.733756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35265 (local address 127.0.97.1:40853)
0621 02:38:20.734632 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.734646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.734795 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.735095 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.735100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.735122 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:20.735443 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.735456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.736461 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.736466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.737845 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.737856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.738928 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.738933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.739082 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.739104 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.739216 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.739586 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":551,"threads_started":1}
I20260621 02:38:20.740460 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74fceaa0c3246a39de22e208e627be2" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68fddd4fd494c32ad9846e399193e15" is_pre_election: true
I20260621 02:38:20.740932 2433 raft_consensus.cc:2454] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 0.
I20260621 02:38:20.740868 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74fceaa0c3246a39de22e208e627be2" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39" is_pre_election: true
I20260621 02:38:20.741423 2572 raft_consensus.cc:2454] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 0.
I20260621 02:38:20.741672 2222 leader_election.cc:302] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2501fe77d7e49199da3ae1f5ba9cc88, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:20.742079 2644 raft_consensus.cc:2781] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:20.742259 2644 raft_consensus.cc:482] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:20.742370 2644 raft_consensus.cc:3037] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:20.744467 2644 raft_consensus.cc:504] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } }
I20260621 02:38:20.745002 2644 leader_election.cc:288] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 election: Requested vote from peers f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:20.745462 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74fceaa0c3246a39de22e208e627be2" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68fddd4fd494c32ad9846e399193e15"
I20260621 02:38:20.745568 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74fceaa0c3246a39de22e208e627be2" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39"
I20260621 02:38:20.745662 2433 raft_consensus.cc:3037] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:20.745786 2572 raft_consensus.cc:3037] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:20.747767 2433 raft_consensus.cc:2454] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 1.
I20260621 02:38:20.747857 2572 raft_consensus.cc:2454] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 1.
I20260621 02:38:20.748226 2222 leader_election.cc:302] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2501fe77d7e49199da3ae1f5ba9cc88, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:20.748476 2644 raft_consensus.cc:2781] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:20.748871 2644 raft_consensus.cc:686] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 LEADER]: Becoming Leader. State: Replica: d2501fe77d7e49199da3ae1f5ba9cc88, State: Running, Role: LEADER
I20260621 02:38:20.749331 2644 consensus_queue.cc:237] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } }
I20260621 02:38:20.753943 2131 catalog_manager.cc:5495] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } health_report { overall_health: HEALTHY } } }
W20260621 02:38:20.776125 2340 tablet.cc:2369] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:38:20 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:20.812471 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.799483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.0.97.62:42405)
0621 02:38:20.799668 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.799672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.800298 (+ 626us) server_negotiation.cc:407] Connection header received
0621 02:38:20.800491 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.800497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.800559 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.800654 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:20.801858 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.802563 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.803789 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.804211 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.804346 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.807458 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:20.807464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:20.807467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:20.807492 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:20.809970 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.810605 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.810610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.810614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.810736 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:20.811094 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:20.811101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:20.811104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:20.811343 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:20.811490 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.811868 (+ 378us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.812172 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":77}
I20260621 02:38:20.812502 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.799523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42405 (local address 127.0.0.1:58628)
0621 02:38:20.800094 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.800115 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.800370 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.800674 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.800678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.801455 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:20.801723 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.801732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.802686 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.802694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.803646 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.803656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.804383 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.804389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.804564 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.805164 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:20.805180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:20.807288 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:20.810090 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.810103 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.810111 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.810481 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.810837 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:20.810843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:20.810846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:20.810976 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:20.811478 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:20.811485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:20.811635 (+ 150us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.811794 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.812065 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":308,"threads_started":1}
I20260621 02:38:20.817370 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58628:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 02:38:20.818269 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260621 02:38:20.822566 2479 tablet.cc:2369] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:38:20.823344 2618 tablet.cc:2369] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:38:20.824674 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 02eddaf6507f4474b4956430d2533768 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27c6e6ac33f747fcba1d7c6bb2301eeb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:38:20.825054 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 02eddaf6507f4474b4956430d2533768 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27c6e6ac33f747fcba1d7c6bb2301eeb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:38:20.825683 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02eddaf6507f4474b4956430d2533768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:20.825278 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 02eddaf6507f4474b4956430d2533768 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27c6e6ac33f747fcba1d7c6bb2301eeb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:38:20.825968 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02eddaf6507f4474b4956430d2533768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:20.826148 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02eddaf6507f4474b4956430d2533768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:20.831241 2636 tablet_bootstrap.cc:492] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: Bootstrap starting.
I20260621 02:38:20.832831 2639 tablet_bootstrap.cc:492] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: Bootstrap starting.
I20260621 02:38:20.832901 2636 tablet_bootstrap.cc:654] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:20.834316 2639 tablet_bootstrap.cc:654] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:20.834357 2638 tablet_bootstrap.cc:492] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: Bootstrap starting.
I20260621 02:38:20.835371 2636 tablet_bootstrap.cc:492] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: No bootstrap required, opened a new log
I20260621 02:38:20.835561 2636 ts_tablet_manager.cc:1397] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260621 02:38:20.836036 2638 tablet_bootstrap.cc:654] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:20.836346 2636 raft_consensus.cc:348] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:20.836576 2639 tablet_bootstrap.cc:492] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: No bootstrap required, opened a new log
I20260621 02:38:20.836562 2636 raft_consensus.cc:374] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:20.836671 2636 raft_consensus.cc:729] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68fddd4fd494c32ad9846e399193e15, State: Initialized, Role: FOLLOWER
I20260621 02:38:20.836715 2639 ts_tablet_manager.cc:1397] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 02:38:20.837001 2636 consensus_queue.cc:260] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:20.837488 2639 raft_consensus.cc:348] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:20.837711 2639 raft_consensus.cc:374] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:20.837533 2642 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.837797 2639 raft_consensus.cc:729] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8b00ef016942a181ecac06da99bb39, State: Initialized, Role: FOLLOWER
I20260621 02:38:20.837909 2642 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:20.838176 2638 tablet_bootstrap.cc:492] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: No bootstrap required, opened a new log
I20260621 02:38:20.838310 2638 ts_tablet_manager.cc:1397] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260621 02:38:20.838446 2639 consensus_queue.cc:260] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:20.838881 2643 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.839195 2643 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:20.839056 2638 raft_consensus.cc:348] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:20.839282 2638 raft_consensus.cc:374] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:20.839385 2638 raft_consensus.cc:729] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2501fe77d7e49199da3ae1f5ba9cc88, State: Initialized, Role: FOLLOWER
I20260621 02:38:20.839622 2638 consensus_queue.cc:260] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:20.839947 2636 ts_tablet_manager.cc:1428] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260621 02:38:20.840224 2638 ts_tablet_manager.cc:1428] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 02:38:20.841010 2639 ts_tablet_manager.cc:1428] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260621 02:38:20.840097 2644 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.841471 2644 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:20.846909 2644 raft_consensus.cc:482] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:20.847097 2644 raft_consensus.cc:504] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:20.847644 2644 leader_election.cc:288] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:20.848094 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02eddaf6507f4474b4956430d2533768" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68fddd4fd494c32ad9846e399193e15" is_pre_election: true
I20260621 02:38:20.848160 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02eddaf6507f4474b4956430d2533768" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39" is_pre_election: true
I20260621 02:38:20.848304 2433 raft_consensus.cc:2454] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 0.
I20260621 02:38:20.848443 2572 raft_consensus.cc:2454] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 0.
I20260621 02:38:20.848701 2222 leader_election.cc:302] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2501fe77d7e49199da3ae1f5ba9cc88, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:20.848951 2644 raft_consensus.cc:2781] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:20.849047 2644 raft_consensus.cc:482] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:20.849157 2644 raft_consensus.cc:3037] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:20.851234 2644 raft_consensus.cc:504] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:20.851717 2644 leader_election.cc:288] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 election: Requested vote from peers f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:20.852087 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02eddaf6507f4474b4956430d2533768" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68fddd4fd494c32ad9846e399193e15"
I20260621 02:38:20.852167 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02eddaf6507f4474b4956430d2533768" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39"
I20260621 02:38:20.852279 2433 raft_consensus.cc:3037] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:20.852370 2572 raft_consensus.cc:3037] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:20.854233 2433 raft_consensus.cc:2454] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 1.
I20260621 02:38:20.854307 2572 raft_consensus.cc:2454] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 1.
I20260621 02:38:20.854588 2222 leader_election.cc:302] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2501fe77d7e49199da3ae1f5ba9cc88, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:20.854846 2644 raft_consensus.cc:2781] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:20.855013 2644 raft_consensus.cc:686] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 LEADER]: Becoming Leader. State: Replica: d2501fe77d7e49199da3ae1f5ba9cc88, State: Running, Role: LEADER
I20260621 02:38:20.855225 2644 consensus_queue.cc:237] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:20.856779 2647 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: TxnStatusTablet state changed. Reason: New leader d2501fe77d7e49199da3ae1f5ba9cc88. Latest consensus state: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.857174 2647 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:38:20.857909 2659 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 02:38:20.858201 2659 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 02:38:20.858243 2131 catalog_manager.cc:5495] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } health_report { overall_health: UNKNOWN } } }
I20260621 02:38:20.879935 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.874203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.0.97.1:34105)
0621 02:38:20.874389 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.874395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.874549 (+ 154us) server_negotiation.cc:407] Connection header received
0621 02:38:20.874817 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.874828 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.874910 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.875066 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:20.875426 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.876377 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.877868 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.878287 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.878422 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.878864 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:20.879317 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:20.879477 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.879533 (+ 56us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.879623 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:38:20.879931 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.874241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34105 (local address 127.0.0.1:49406)
0621 02:38:20.874454 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.874470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.874620 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.875104 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.875112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.875123 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:38:20.875301 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.875309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.876492 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.876500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.877731 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.877742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.878431 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.878436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.878623 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.878670 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:38:20.879427 (+ 757us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:38:20.879445 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.879517 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.879653 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 02:38:20.887217 2572 raft_consensus.cc:1261] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Refusing update from remote peer d2501fe77d7e49199da3ae1f5ba9cc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:20.887349 2433 raft_consensus.cc:1261] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Refusing update from remote peer d2501fe77d7e49199da3ae1f5ba9cc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:20.888123 2647 consensus_queue.cc:1035] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:20.888741 2644 consensus_queue.cc:1035] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:20.901350 2642 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: TxnStatusTablet state changed. Reason: New leader d2501fe77d7e49199da3ae1f5ba9cc88. Latest consensus state: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.901634 2642 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:20.905895 2643 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: TxnStatusTablet state changed. Reason: New leader d2501fe77d7e49199da3ae1f5ba9cc88. Latest consensus state: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.906188 2643 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:20.909495 2647 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.909807 2647 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:38:20.913677 2642 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.913892 2642 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P f68fddd4fd494c32ad9846e399193e15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:20.916976 2644 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.917248 2644 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P d2501fe77d7e49199da3ae1f5ba9cc88: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:38:20.918962 2663 mvcc.cc:205] Tried to move back new op lower bound from 7299110915625205760 to 7299110915504820224. Current Snapshot: MvccSnapshot[applied={T|T < 7299110915625205760}]
I20260621 02:38:20.919456 2643 tablet_replica.cc:439] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } } }
I20260621 02:38:20.919675 2643 tablet_replica.cc:442] T 02eddaf6507f4474b4956430d2533768 P 5b8b00ef016942a181ecac06da99bb39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:38:20.928186 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.922801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.0.97.1:34105)
0621 02:38:20.922979 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.922985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.923188 (+ 203us) server_negotiation.cc:407] Connection header received
0621 02:38:20.923359 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.923362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.923432 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.923503 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:20.923894 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.924611 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.926514 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.926919 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.927013 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.927483 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:20.927599 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:20.927663 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.927836 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.927901 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:38:20.946396 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.939412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37493 (local address 127.0.97.1:34105)
0621 02:38:20.939573 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:20.939579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:20.940156 (+ 577us) server_negotiation.cc:407] Connection header received
0621 02:38:20.940323 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:20.940328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:20.940394 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:20.940477 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:20.941097 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.942283 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.944383 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.945064 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.945179 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.945716 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:20.945827 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:20.945909 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:20.946051 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 02:38:20.946119 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 02:38:20.946774 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:20.939390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34105 (local address 127.0.97.1:37493)
0621 02:38:20.940051 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:20.940073 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:20.940206 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:20.940513 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:20.940518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:20.940530 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:38:20.940916 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.940927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.942409 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.942414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.944238 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:20.944251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:20.945245 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:20.945252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:20.945455 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:20.945532 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:38:20.945926 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:38:20.945936 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:38:20.946017 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 02:38:20.946145 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":4,"thread_start_us":345,"threads_started":1}
I20260621 02:38:20.956115 2433 raft_consensus.cc:1261] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Refusing update from remote peer d2501fe77d7e49199da3ae1f5ba9cc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:20.956184 2572 raft_consensus.cc:1261] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Refusing update from remote peer d2501fe77d7e49199da3ae1f5ba9cc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:20.956668 2647 consensus_queue.cc:1035] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:20.956923 2644 consensus_queue.cc:1035] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:20.961347 2662 mvcc.cc:205] Tried to move back new op lower bound from 7299110915911843840 to 7299110915072405504. Current Snapshot: MvccSnapshot[applied={T|T < 7299110915911843840}]
I20260621 02:38:21.015583 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:21.015884 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:21.019524 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237: Sending DeleteTablet for 3 replicas of tablet e74fceaa0c3246a39de22e208e627be2
I20260621 02:38:21.020457 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet e74fceaa0c3246a39de22e208e627be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35442
I20260621 02:38:21.020637 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet e74fceaa0c3246a39de22e208e627be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47430
I20260621 02:38:21.022503 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet e74fceaa0c3246a39de22e208e627be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49394
I20260621 02:38:21.022480 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:38:21.023104 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:21.032579 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 79eef4b01fe1448b94783fb9c6c21508 (DEFAULT_TABLE table=test-table [id=d9c1bd74852048b988e5a2d12799629d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:21.032639 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 79eef4b01fe1448b94783fb9c6c21508 (DEFAULT_TABLE table=test-table [id=d9c1bd74852048b988e5a2d12799629d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:21.033135 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79eef4b01fe1448b94783fb9c6c21508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:21.033243 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79eef4b01fe1448b94783fb9c6c21508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:21.035917 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 79eef4b01fe1448b94783fb9c6c21508 (DEFAULT_TABLE table=test-table [id=d9c1bd74852048b988e5a2d12799629d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:21.036435 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79eef4b01fe1448b94783fb9c6c21508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:21.036669 2679 tablet_replica.cc:330] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: stopping tablet replica
I20260621 02:38:21.037237 2679 raft_consensus.cc:2229] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:21.038233 2679 raft_consensus.cc:2258] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:21.039613 2636 tablet_bootstrap.cc:492] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: Bootstrap starting.
I20260621 02:38:21.041239 2636 tablet_bootstrap.cc:654] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:21.043951 2678 tablet_replica.cc:330] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: stopping tablet replica
I20260621 02:38:21.044345 2678 raft_consensus.cc:2229] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:21.044688 2678 raft_consensus.cc:2258] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:21.046658 2678 ts_tablet_manager.cc:1905] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:21.053305 2639 tablet_bootstrap.cc:492] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: Bootstrap starting.
I20260621 02:38:21.053959 2678 ts_tablet_manager.cc:1918] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:38:21.054098 2678 log.cc:1196] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/e74fceaa0c3246a39de22e208e627be2
I20260621 02:38:21.054579 2678 ts_tablet_manager.cc:1939] T e74fceaa0c3246a39de22e208e627be2 P f68fddd4fd494c32ad9846e399193e15: Deleting consensus metadata
I20260621 02:38:21.054834 2639 tablet_bootstrap.cc:654] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:21.055675 2638 tablet_bootstrap.cc:492] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: Bootstrap starting.
I20260621 02:38:21.056025 2680 tablet_replica.cc:330] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: stopping tablet replica
I20260621 02:38:21.056305 2119 catalog_manager.cc:4841] TS f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587): tablet e74fceaa0c3246a39de22e208e627be2 (table test-table [id=c3fdd900156047afa335720d37be0147]) successfully deleted
I20260621 02:38:21.056452 2680 raft_consensus.cc:2229] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:38:21.056995 2680 raft_consensus.cc:2258] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:21.057881 2638 tablet_bootstrap.cc:654] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:21.058967 2680 ts_tablet_manager.cc:1905] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:21.062361 2636 tablet_bootstrap.cc:492] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: No bootstrap required, opened a new log
I20260621 02:38:21.062778 2636 ts_tablet_manager.cc:1397] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20260621 02:38:21.064106 2636 raft_consensus.cc:348] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:21.065272 2636 raft_consensus.cc:374] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:21.065912 2636 raft_consensus.cc:729] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68fddd4fd494c32ad9846e399193e15, State: Initialized, Role: FOLLOWER
I20260621 02:38:21.066737 2680 ts_tablet_manager.cc:1918] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:38:21.066890 2680 log.cc:1196] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/e74fceaa0c3246a39de22e208e627be2
I20260621 02:38:21.067462 2680 ts_tablet_manager.cc:1939] T e74fceaa0c3246a39de22e208e627be2 P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting consensus metadata
I20260621 02:38:21.067831 2639 tablet_bootstrap.cc:492] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: No bootstrap required, opened a new log
I20260621 02:38:21.068063 2639 ts_tablet_manager.cc:1397] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260621 02:38:21.069108 2116 catalog_manager.cc:4841] TS d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1:34105): tablet e74fceaa0c3246a39de22e208e627be2 (table test-table [id=c3fdd900156047afa335720d37be0147]) successfully deleted
I20260621 02:38:21.069005 2639 raft_consensus.cc:348] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:21.069397 2639 raft_consensus.cc:374] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:21.069485 2639 raft_consensus.cc:729] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8b00ef016942a181ecac06da99bb39, State: Initialized, Role: FOLLOWER
I20260621 02:38:21.069705 2639 consensus_queue.cc:260] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:21.070341 2639 ts_tablet_manager.cc:1428] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 02:38:21.066708 2636 consensus_queue.cc:260] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:21.073693 2636 ts_tablet_manager.cc:1428] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260621 02:38:21.075947 2679 ts_tablet_manager.cc:1905] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:21.083871 2679 ts_tablet_manager.cc:1918] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:38:21.084033 2679 log.cc:1196] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/e74fceaa0c3246a39de22e208e627be2
I20260621 02:38:21.084452 2679 ts_tablet_manager.cc:1939] T e74fceaa0c3246a39de22e208e627be2 P 5b8b00ef016942a181ecac06da99bb39: Deleting consensus metadata
I20260621 02:38:21.086059 2119 catalog_manager.cc:4841] TS 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265): tablet e74fceaa0c3246a39de22e208e627be2 (table test-table [id=c3fdd900156047afa335720d37be0147]) successfully deleted
I20260621 02:38:21.089846 2638 tablet_bootstrap.cc:492] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: No bootstrap required, opened a new log
I20260621 02:38:21.090039 2638 ts_tablet_manager.cc:1397] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.008s
I20260621 02:38:21.090662 2638 raft_consensus.cc:348] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:21.090878 2638 raft_consensus.cc:374] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:21.090991 2638 raft_consensus.cc:729] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2501fe77d7e49199da3ae1f5ba9cc88, State: Initialized, Role: FOLLOWER
I20260621 02:38:21.091264 2638 consensus_queue.cc:260] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:21.091789 2638 ts_tablet_manager.cc:1428] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 02:38:21.227725 2642 raft_consensus.cc:482] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:21.227977 2642 raft_consensus.cc:504] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:21.229998 2642 leader_election.cc:288] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1:34105), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:21.237985 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:21.230382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42695 (local address 127.0.97.3:35265)
0621 02:38:21.230590 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:21.230599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:21.231459 (+ 860us) server_negotiation.cc:407] Connection header received
0621 02:38:21.232015 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:21.232022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:21.232105 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:21.232204 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:21.232792 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.233886 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.236224 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.236964 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.237113 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:21.237213 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:21.237575 (+ 362us) server_negotiation.cc:299] Negotiation successful
0621 02:38:21.237716 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:38:21.238058 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:21.230763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35265 (local address 127.0.97.2:42695)
0621 02:38:21.231365 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:21.231379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:21.231473 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:21.232241 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:21.232245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:21.232274 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:21.232632 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:21.232643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.234025 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.234030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:21.236090 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:21.236100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.237165 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.237171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:21.237403 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:21.237438 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:38:21.237533 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:38:21.237669 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":23,"thread_start_us":326,"threads_started":1}
I20260621 02:38:21.238142 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:21.230407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38219 (local address 127.0.97.1:34105)
0621 02:38:21.230634 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:21.230643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:21.231346 (+ 703us) server_negotiation.cc:407] Connection header received
0621 02:38:21.231615 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:21.231623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:21.231704 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:21.231826 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:21.232319 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.233225 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.235705 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.237043 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.237178 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:21.237309 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:21.237728 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 02:38:21.237863 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:38:21.238770 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:21.230319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34105 (local address 127.0.97.2:38219)
0621 02:38:21.231218 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:21.231243 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:21.231386 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:21.231835 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:21.231838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:21.231857 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:21.232119 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:21.232127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.233428 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.233431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:21.235508 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:21.235521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.237385 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.237392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:21.237560 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:21.237588 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 02:38:21.238291 (+ 703us) client_negotiation.cc:240] Negotiation successful
0621 02:38:21.238406 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":1,"thread_start_us":475,"threads_started":1}
I20260621 02:38:21.239024 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eef4b01fe1448b94783fb9c6c21508" candidate_uuid: "f68fddd4fd494c32ad9846e399193e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39" is_pre_election: true
I20260621 02:38:21.239264 2572 raft_consensus.cc:2454] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68fddd4fd494c32ad9846e399193e15 in term 0.
I20260621 02:38:21.239730 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eef4b01fe1448b94783fb9c6c21508" candidate_uuid: "f68fddd4fd494c32ad9846e399193e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" is_pre_election: true
I20260621 02:38:21.240027 2362 leader_election.cc:302] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b8b00ef016942a181ecac06da99bb39, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:21.240113 2289 raft_consensus.cc:2454] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68fddd4fd494c32ad9846e399193e15 in term 0.
I20260621 02:38:21.240440 2642 raft_consensus.cc:2781] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:21.240561 2642 raft_consensus.cc:482] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:21.240664 2642 raft_consensus.cc:3037] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:21.242980 2642 raft_consensus.cc:504] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:21.243512 2642 leader_election.cc:288] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [CANDIDATE]: Term 1 election: Requested vote from peers d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1:34105), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:21.243949 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eef4b01fe1448b94783fb9c6c21508" candidate_uuid: "f68fddd4fd494c32ad9846e399193e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88"
I20260621 02:38:21.243983 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eef4b01fe1448b94783fb9c6c21508" candidate_uuid: "f68fddd4fd494c32ad9846e399193e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39"
I20260621 02:38:21.244158 2289 raft_consensus.cc:3037] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:21.244203 2572 raft_consensus.cc:3037] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:21.246271 2289 raft_consensus.cc:2454] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68fddd4fd494c32ad9846e399193e15 in term 1.
I20260621 02:38:21.246662 2359 leader_election.cc:302] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2501fe77d7e49199da3ae1f5ba9cc88, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:21.246923 2642 raft_consensus.cc:2781] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:21.247169 2572 raft_consensus.cc:2454] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68fddd4fd494c32ad9846e399193e15 in term 1.
I20260621 02:38:21.247826 2642 raft_consensus.cc:686] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 1 LEADER]: Becoming Leader. State: Replica: f68fddd4fd494c32ad9846e399193e15, State: Running, Role: LEADER
I20260621 02:38:21.248618 2642 consensus_queue.cc:237] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:21.252118 2132 catalog_manager.cc:5495] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 reported cstate change: term changed from 0 to 1, leader changed from <none> to f68fddd4fd494c32ad9846e399193e15 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f68fddd4fd494c32ad9846e399193e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } health_report { overall_health: UNKNOWN } } }
I20260621 02:38:21.312668 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:21.306519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.0.97.2:37587)
0621 02:38:21.307572 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:21.307575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:21.307592 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:38:21.307669 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:21.307672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:21.307735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:21.307900 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:21.308264 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.309086 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.310565 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.310970 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.311106 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:21.311511 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:21.312034 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:21.312189 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:21.312270 (+ 81us) server_negotiation.cc:299] Negotiation successful
0621 02:38:21.312341 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":835,"threads_started":1}
I20260621 02:38:21.327826 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:21.322296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46857 (local address 127.0.97.1:34105)
0621 02:38:21.322473 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:21.322481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:21.322910 (+ 429us) server_negotiation.cc:407] Connection header received
0621 02:38:21.323105 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:21.323109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:21.323176 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:21.323270 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:21.323701 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.324567 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.326116 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.326523 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.326659 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:21.327094 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:21.327216 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:21.327312 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:21.327440 (+ 128us) server_negotiation.cc:299] Negotiation successful
0621 02:38:21.327515 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:38:21.327907 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:21.322224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34105 (local address 127.0.97.2:46857)
0621 02:38:21.322815 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:21.322830 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:21.322972 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:21.323282 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:21.323287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:21.323298 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:38:21.323558 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:21.323565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.324705 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.324711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:21.325971 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:21.325982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.326737 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.326742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:21.326865 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:21.326912 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:38:21.327312 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:38:21.327324 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 02:38:21.327418 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 02:38:21.327537 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":332,"threads_started":1}
I20260621 02:38:21.337154 2572 raft_consensus.cc:1261] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Refusing update from remote peer f68fddd4fd494c32ad9846e399193e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:21.337507 2289 raft_consensus.cc:1261] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Refusing update from remote peer f68fddd4fd494c32ad9846e399193e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:21.337747 2686 consensus_queue.cc:1035] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:21.338214 2642 consensus_queue.cc:1035] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:21.342278 2664 mvcc.cc:205] Tried to move back new op lower bound from 7299110917471240192 to 7299110917118210048. Current Snapshot: MvccSnapshot[applied={T|T < 7299110917471240192}]
I20260621 02:38:21.370451 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:21.364893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52951 (local address 127.0.97.2:37587)
0621 02:38:21.365043 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:21.365050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:21.365307 (+ 257us) server_negotiation.cc:407] Connection header received
0621 02:38:21.365501 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:21.365506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:21.365589 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:21.365683 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:21.366092 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.367117 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.368421 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.368821 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.368952 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:21.369700 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:21.369822 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:21.369914 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:21.370043 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 02:38:21.370120 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:38:21.370481 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:21.364947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37587 (local address 127.0.97.1:52951)
0621 02:38:21.365195 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:21.365210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:21.365366 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:21.365726 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:21.365732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:21.365745 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:38:21.365940 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:21.365950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.367230 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.367238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:21.368283 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:21.368295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:21.368980 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:21.368986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:21.369164 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:21.369227 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:38:21.369917 (+ 690us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:38:21.369930 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 02:38:21.370028 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:38:21.370178 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
W20260621 02:38:21.371472 2664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 02:38:22.981043 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:22.981240 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:22.985145 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237: Sending DeleteTablet for 3 replicas of tablet 79eef4b01fe1448b94783fb9c6c21508
I20260621 02:38:22.985922 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 79eef4b01fe1448b94783fb9c6c21508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49394
I20260621 02:38:22.986047 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 79eef4b01fe1448b94783fb9c6c21508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35442
I20260621 02:38:22.986171 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 79eef4b01fe1448b94783fb9c6c21508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47430
I20260621 02:38:22.987344 2718 tablet_replica.cc:330] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: stopping tablet replica
I20260621 02:38:22.987656 2718 raft_consensus.cc:2229] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:22.987989 2718 raft_consensus.cc:2258] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:22.989250 2718 ts_tablet_manager.cc:1905] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:22.998595 2718 ts_tablet_manager.cc:1918] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:38:22.998864 2718 log.cc:1196] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/79eef4b01fe1448b94783fb9c6c21508
I20260621 02:38:22.999495 2718 ts_tablet_manager.cc:1939] T 79eef4b01fe1448b94783fb9c6c21508 P 5b8b00ef016942a181ecac06da99bb39: Deleting consensus metadata
I20260621 02:38:23.000921 2119 catalog_manager.cc:4841] TS 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265): tablet 79eef4b01fe1448b94783fb9c6c21508 (table test-table [id=d9c1bd74852048b988e5a2d12799629d]) successfully deleted
I20260621 02:38:23.001183 2716 tablet_replica.cc:330] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: stopping tablet replica
I20260621 02:38:23.001263 2717 tablet_replica.cc:330] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: stopping tablet replica
I20260621 02:38:23.001401 2716 raft_consensus.cc:2229] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:23.001521 2717 raft_consensus.cc:2229] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:38:23.001684 2716 raft_consensus.cc:2258] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:23.002038 2717 raft_consensus.cc:2258] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:23.002564 2716 ts_tablet_manager.cc:1905] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:23.002966 2717 ts_tablet_manager.cc:1905] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:23.007241 2716 ts_tablet_manager.cc:1918] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:38:23.007391 2716 log.cc:1196] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/79eef4b01fe1448b94783fb9c6c21508
I20260621 02:38:23.007781 2716 ts_tablet_manager.cc:1939] T 79eef4b01fe1448b94783fb9c6c21508 P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting consensus metadata
I20260621 02:38:23.008870 2116 catalog_manager.cc:4841] TS d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1:34105): tablet 79eef4b01fe1448b94783fb9c6c21508 (table test-table [id=d9c1bd74852048b988e5a2d12799629d]) successfully deleted
I20260621 02:38:23.010075 2717 ts_tablet_manager.cc:1918] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:38:23.010174 2717 log.cc:1196] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/79eef4b01fe1448b94783fb9c6c21508
I20260621 02:38:23.010613 2717 ts_tablet_manager.cc:1939] T 79eef4b01fe1448b94783fb9c6c21508 P f68fddd4fd494c32ad9846e399193e15: Deleting consensus metadata
I20260621 02:38:23.011497 2119 catalog_manager.cc:4841] TS f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587): tablet 79eef4b01fe1448b94783fb9c6c21508 (table test-table [id=d9c1bd74852048b988e5a2d12799629d]) successfully deleted
Jun 21 02:38:23 dist-test-slave-jkn7 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:38:23 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:23.047844 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:23.029680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.0.97.62:42405)
0621 02:38:23.030214 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:23.030218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:23.030241 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:38:23.030393 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:23.030397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:23.030459 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:23.030558 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:23.031800 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:23.032564 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:23.034118 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:23.034962 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:23.035132 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:23.042771 (+ 7639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:23.042777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:23.042780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:23.042813 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:23.045323 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:23.045948 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:23.045953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:23.045956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:23.046079 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:23.046453 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:23.046458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:23.046461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:23.046727 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:23.046965 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:23.047263 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 02:38:23.047476 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":286,"threads_started":1}
I20260621 02:38:23.050998 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:38:23.051594 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:23.058017 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 0469bacf3f7e4778b13b75c69b61f1f3 (DEFAULT_TABLE table=test-table [id=0a36c2c69ec94f13ad72e31627ffa773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:23.058045 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 0469bacf3f7e4778b13b75c69b61f1f3 (DEFAULT_TABLE table=test-table [id=0a36c2c69ec94f13ad72e31627ffa773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:23.058166 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 0469bacf3f7e4778b13b75c69b61f1f3 (DEFAULT_TABLE table=test-table [id=0a36c2c69ec94f13ad72e31627ffa773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:23.058463 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0469bacf3f7e4778b13b75c69b61f1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:23.058472 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0469bacf3f7e4778b13b75c69b61f1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:23.058564 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0469bacf3f7e4778b13b75c69b61f1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:23.064098 2727 tablet_bootstrap.cc:492] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: Bootstrap starting.
I20260621 02:38:23.065405 2728 tablet_bootstrap.cc:492] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: Bootstrap starting.
I20260621 02:38:23.065804 2727 tablet_bootstrap.cc:654] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:23.066133 2729 tablet_bootstrap.cc:492] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: Bootstrap starting.
I20260621 02:38:23.067318 2728 tablet_bootstrap.cc:654] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:23.068162 2729 tablet_bootstrap.cc:654] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:23.069319 2727 tablet_bootstrap.cc:492] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: No bootstrap required, opened a new log
I20260621 02:38:23.069484 2727 ts_tablet_manager.cc:1397] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260621 02:38:23.070118 2729 tablet_bootstrap.cc:492] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: No bootstrap required, opened a new log
I20260621 02:38:23.070276 2729 ts_tablet_manager.cc:1397] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 02:38:23.070250 2727 raft_consensus.cc:348] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.070480 2727 raft_consensus.cc:374] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:23.070565 2727 raft_consensus.cc:729] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68fddd4fd494c32ad9846e399193e15, State: Initialized, Role: FOLLOWER
I20260621 02:38:23.070765 2727 consensus_queue.cc:260] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.071130 2729 raft_consensus.cc:348] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.071291 2727 ts_tablet_manager.cc:1428] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 02:38:23.071357 2729 raft_consensus.cc:374] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:23.071452 2729 raft_consensus.cc:729] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2501fe77d7e49199da3ae1f5ba9cc88, State: Initialized, Role: FOLLOWER
I20260621 02:38:23.071722 2729 consensus_queue.cc:260] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.072234 2729 ts_tablet_manager.cc:1428] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:38:23.072654 2728 tablet_bootstrap.cc:492] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: No bootstrap required, opened a new log
I20260621 02:38:23.072779 2728 ts_tablet_manager.cc:1397] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260621 02:38:23.073787 2728 raft_consensus.cc:348] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.074002 2728 raft_consensus.cc:374] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:23.074084 2728 raft_consensus.cc:729] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8b00ef016942a181ecac06da99bb39, State: Initialized, Role: FOLLOWER
I20260621 02:38:23.074380 2728 consensus_queue.cc:260] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.081808 2728 ts_tablet_manager.cc:1428] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260621 02:38:23.383839 2647 raft_consensus.cc:482] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:23.384034 2647 raft_consensus.cc:504] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.384601 2647 leader_election.cc:288] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:23.385314 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0469bacf3f7e4778b13b75c69b61f1f3" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39" is_pre_election: true
I20260621 02:38:23.385367 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0469bacf3f7e4778b13b75c69b61f1f3" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68fddd4fd494c32ad9846e399193e15" is_pre_election: true
I20260621 02:38:23.385571 2572 raft_consensus.cc:2454] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 0.
I20260621 02:38:23.385603 2432 raft_consensus.cc:2454] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 0.
I20260621 02:38:23.386067 2222 leader_election.cc:302] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2501fe77d7e49199da3ae1f5ba9cc88, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:23.386327 2647 raft_consensus.cc:2781] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:23.386447 2647 raft_consensus.cc:482] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:23.386543 2647 raft_consensus.cc:3037] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:23.388473 2647 raft_consensus.cc:504] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.389436 2647 leader_election.cc:288] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 election: Requested vote from peers f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:23.389523 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0469bacf3f7e4778b13b75c69b61f1f3" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68fddd4fd494c32ad9846e399193e15"
I20260621 02:38:23.389742 2432 raft_consensus.cc:3037] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:23.389919 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0469bacf3f7e4778b13b75c69b61f1f3" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39"
I20260621 02:38:23.390122 2572 raft_consensus.cc:3037] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:23.392766 2432 raft_consensus.cc:2454] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 1.
I20260621 02:38:23.392936 2572 raft_consensus.cc:2454] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 1.
I20260621 02:38:23.393178 2222 leader_election.cc:302] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2501fe77d7e49199da3ae1f5ba9cc88, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:23.393499 2647 raft_consensus.cc:2781] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:23.393673 2647 raft_consensus.cc:686] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 LEADER]: Becoming Leader. State: Replica: d2501fe77d7e49199da3ae1f5ba9cc88, State: Running, Role: LEADER
I20260621 02:38:23.393939 2647 consensus_queue.cc:237] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.396345 2132 catalog_manager.cc:5495] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } health_report { overall_health: UNKNOWN } } }
I20260621 02:38:23.443141 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:23.437775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.0.97.1:34105)
0621 02:38:23.438327 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:23.438331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:23.438350 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:38:23.438396 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:23.438399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:23.438458 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:23.438550 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:23.438839 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:23.439573 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:23.441461 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:23.441952 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:23.442049 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:23.442440 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:23.442545 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:23.442615 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:23.442744 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 02:38:23.442798 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":362,"threads_started":1}
I20260621 02:38:23.451865 2432 raft_consensus.cc:1261] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Refusing update from remote peer d2501fe77d7e49199da3ae1f5ba9cc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:23.451848 2572 raft_consensus.cc:1261] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Refusing update from remote peer d2501fe77d7e49199da3ae1f5ba9cc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:23.452359 2647 consensus_queue.cc:1035] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:23.452683 2703 consensus_queue.cc:1035] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:23.498438 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:23.498683 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:23.501405 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237: Sending DeleteTablet for 3 replicas of tablet 0469bacf3f7e4778b13b75c69b61f1f3
I20260621 02:38:23.502127 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 0469bacf3f7e4778b13b75c69b61f1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35442
I20260621 02:38:23.502266 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 0469bacf3f7e4778b13b75c69b61f1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47430
I20260621 02:38:23.502408 2717 tablet_replica.cc:330] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: stopping tablet replica
I20260621 02:38:23.502661 2717 raft_consensus.cc:2229] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:23.502106 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 0469bacf3f7e4778b13b75c69b61f1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49394
I20260621 02:38:23.502878 2717 raft_consensus.cc:2258] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:23.503113 2716 tablet_replica.cc:330] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: stopping tablet replica
I20260621 02:38:23.503139 2742 tablet_replica.cc:330] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: stopping tablet replica
I20260621 02:38:23.503310 2716 raft_consensus.cc:2229] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:38:23.503337 2742 raft_consensus.cc:2229] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:23.503290 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 02:38:23.503520 2742 raft_consensus.cc:2258] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:23.503597 2716 raft_consensus.cc:2258] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 02:38:23.503913 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:23.504043 2717 ts_tablet_manager.cc:1905] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:23.504598 2716 ts_tablet_manager.cc:1905] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:23.504608 2742 ts_tablet_manager.cc:1905] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:23.509335 2742 ts_tablet_manager.cc:1918] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:38:23.509473 2742 log.cc:1196] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/0469bacf3f7e4778b13b75c69b61f1f3
I20260621 02:38:23.509833 2742 ts_tablet_manager.cc:1939] T 0469bacf3f7e4778b13b75c69b61f1f3 P 5b8b00ef016942a181ecac06da99bb39: Deleting consensus metadata
I20260621 02:38:23.510378 2717 ts_tablet_manager.cc:1918] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:38:23.510515 2717 log.cc:1196] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/0469bacf3f7e4778b13b75c69b61f1f3
I20260621 02:38:23.510689 2119 catalog_manager.cc:4841] TS 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265): tablet 0469bacf3f7e4778b13b75c69b61f1f3 (table test-table [id=0a36c2c69ec94f13ad72e31627ffa773]) successfully deleted
I20260621 02:38:23.510793 2716 ts_tablet_manager.cc:1918] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:38:23.510931 2717 ts_tablet_manager.cc:1939] T 0469bacf3f7e4778b13b75c69b61f1f3 P f68fddd4fd494c32ad9846e399193e15: Deleting consensus metadata
I20260621 02:38:23.510932 2716 log.cc:1196] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/0469bacf3f7e4778b13b75c69b61f1f3
I20260621 02:38:23.511324 2716 ts_tablet_manager.cc:1939] T 0469bacf3f7e4778b13b75c69b61f1f3 P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting consensus metadata
I20260621 02:38:23.511904 2119 catalog_manager.cc:4841] TS f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587): tablet 0469bacf3f7e4778b13b75c69b61f1f3 (table test-table [id=0a36c2c69ec94f13ad72e31627ffa773]) successfully deleted
I20260621 02:38:23.512215 2116 catalog_manager.cc:4841] TS d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1:34105): tablet 0469bacf3f7e4778b13b75c69b61f1f3 (table test-table [id=0a36c2c69ec94f13ad72e31627ffa773]) successfully deleted
I20260621 02:38:23.516042 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 56a8c43c0533437997eea0a160ff01fe (DEFAULT_TABLE table=test-table [id=b0128441e3c540f291551cc9c7fef7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:23.516460 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a8c43c0533437997eea0a160ff01fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:23.516749 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 56a8c43c0533437997eea0a160ff01fe (DEFAULT_TABLE table=test-table [id=b0128441e3c540f291551cc9c7fef7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:23.517114 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a8c43c0533437997eea0a160ff01fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:23.517100 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 56a8c43c0533437997eea0a160ff01fe (DEFAULT_TABLE table=test-table [id=b0128441e3c540f291551cc9c7fef7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:23.517489 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a8c43c0533437997eea0a160ff01fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:23.521672 2727 tablet_bootstrap.cc:492] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: Bootstrap starting.
I20260621 02:38:23.523103 2727 tablet_bootstrap.cc:654] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:23.524837 2729 tablet_bootstrap.cc:492] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: Bootstrap starting.
I20260621 02:38:23.526361 2729 tablet_bootstrap.cc:654] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:23.527016 2728 tablet_bootstrap.cc:492] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: Bootstrap starting.
I20260621 02:38:23.527729 2727 tablet_bootstrap.cc:492] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: No bootstrap required, opened a new log
I20260621 02:38:23.527887 2727 ts_tablet_manager.cc:1397] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260621 02:38:23.528625 2728 tablet_bootstrap.cc:654] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:23.528558 2727 raft_consensus.cc:348] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.528784 2727 raft_consensus.cc:374] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:23.528852 2727 raft_consensus.cc:729] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68fddd4fd494c32ad9846e399193e15, State: Initialized, Role: FOLLOWER
I20260621 02:38:23.529088 2727 consensus_queue.cc:260] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.529420 2729 tablet_bootstrap.cc:492] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: No bootstrap required, opened a new log
I20260621 02:38:23.529560 2729 ts_tablet_manager.cc:1397] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 02:38:23.530179 2727 ts_tablet_manager.cc:1428] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:38:23.530318 2729 raft_consensus.cc:348] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.530530 2729 raft_consensus.cc:374] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:23.530611 2729 raft_consensus.cc:729] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2501fe77d7e49199da3ae1f5ba9cc88, State: Initialized, Role: FOLLOWER
I20260621 02:38:23.531023 2729 consensus_queue.cc:260] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.531622 2729 ts_tablet_manager.cc:1428] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:38:23.532902 2728 tablet_bootstrap.cc:492] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: No bootstrap required, opened a new log
I20260621 02:38:23.533059 2728 ts_tablet_manager.cc:1397] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260621 02:38:23.533756 2728 raft_consensus.cc:348] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.533979 2728 raft_consensus.cc:374] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:23.534045 2728 raft_consensus.cc:729] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8b00ef016942a181ecac06da99bb39, State: Initialized, Role: FOLLOWER
I20260621 02:38:23.534243 2728 consensus_queue.cc:260] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.535274 2728 ts_tablet_manager.cc:1428] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:38:23.736800 2703 raft_consensus.cc:482] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:23.737061 2703 raft_consensus.cc:504] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.737702 2703 leader_election.cc:288] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:23.738266 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a8c43c0533437997eea0a160ff01fe" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68fddd4fd494c32ad9846e399193e15" is_pre_election: true
I20260621 02:38:23.738396 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a8c43c0533437997eea0a160ff01fe" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39" is_pre_election: true
I20260621 02:38:23.738502 2432 raft_consensus.cc:2454] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 0.
I20260621 02:38:23.738675 2572 raft_consensus.cc:2454] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 0.
I20260621 02:38:23.738943 2222 leader_election.cc:302] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2501fe77d7e49199da3ae1f5ba9cc88, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:23.739203 2703 raft_consensus.cc:2781] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:23.739305 2703 raft_consensus.cc:482] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:23.739413 2703 raft_consensus.cc:3037] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:23.741472 2703 raft_consensus.cc:504] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.741978 2703 leader_election.cc:288] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 election: Requested vote from peers f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587), 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265)
I20260621 02:38:23.742478 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a8c43c0533437997eea0a160ff01fe" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68fddd4fd494c32ad9846e399193e15"
I20260621 02:38:23.742591 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a8c43c0533437997eea0a160ff01fe" candidate_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8b00ef016942a181ecac06da99bb39"
I20260621 02:38:23.742713 2432 raft_consensus.cc:3037] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:23.742781 2572 raft_consensus.cc:3037] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:23.745992 2432 raft_consensus.cc:2454] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 1.
I20260621 02:38:23.746142 2572 raft_consensus.cc:2454] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2501fe77d7e49199da3ae1f5ba9cc88 in term 1.
I20260621 02:38:23.746456 2222 leader_election.cc:302] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2501fe77d7e49199da3ae1f5ba9cc88, f68fddd4fd494c32ad9846e399193e15; no voters:
I20260621 02:38:23.747203 2703 raft_consensus.cc:2781] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:23.747402 2703 raft_consensus.cc:686] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 LEADER]: Becoming Leader. State: Replica: d2501fe77d7e49199da3ae1f5ba9cc88, State: Running, Role: LEADER
I20260621 02:38:23.747673 2703 consensus_queue.cc:237] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } }
I20260621 02:38:23.750731 2130 catalog_manager.cc:5495] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2501fe77d7e49199da3ae1f5ba9cc88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 } health_report { overall_health: UNKNOWN } } }
I20260621 02:38:23.829058 2572 raft_consensus.cc:1261] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Refusing update from remote peer d2501fe77d7e49199da3ae1f5ba9cc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:23.829174 2432 raft_consensus.cc:1261] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Refusing update from remote peer d2501fe77d7e49199da3ae1f5ba9cc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:23.829622 2697 consensus_queue.cc:1035] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8b00ef016942a181ecac06da99bb39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:23.829963 2703 consensus_queue.cc:1035] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68fddd4fd494c32ad9846e399193e15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:38:23.848351 2695 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 02:38:25.425387 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:25.425604 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:25.429529 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db53966ada334540a3de4ef7ba37c237: Sending DeleteTablet for 3 replicas of tablet 56a8c43c0533437997eea0a160ff01fe
I20260621 02:38:25.430274 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a8c43c0533437997eea0a160ff01fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49394
I20260621 02:38:25.430374 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a8c43c0533437997eea0a160ff01fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35442
I20260621 02:38:25.430389 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a8c43c0533437997eea0a160ff01fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47430
I20260621 02:38:25.431006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2203
I20260621 02:38:25.440799 2769 tablet_replica.cc:330] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: stopping tablet replica
I20260621 02:38:25.441079 2769 raft_consensus.cc:2229] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:25.441347 2769 raft_consensus.cc:2258] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:25.442039 2770 tablet_replica.cc:330] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: stopping tablet replica
I20260621 02:38:25.442233 2768 tablet_replica.cc:330] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: stopping tablet replica
I20260621 02:38:25.442242 2770 raft_consensus.cc:2229] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:25.442410 2768 raft_consensus.cc:2229] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:38:25.442443 2770 raft_consensus.cc:2258] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:25.442494 2769 ts_tablet_manager.cc:1905] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:25.442827 2768 raft_consensus.cc:2258] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:25.443341 2770 ts_tablet_manager.cc:1905] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:25.443727 2768 ts_tablet_manager.cc:1905] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:25.449424 2769 ts_tablet_manager.cc:1918] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:38:25.449563 2769 log.cc:1196] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/56a8c43c0533437997eea0a160ff01fe
I20260621 02:38:25.449635 2770 ts_tablet_manager.cc:1918] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:38:25.449774 2770 log.cc:1196] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/56a8c43c0533437997eea0a160ff01fe
I20260621 02:38:25.449999 2769 ts_tablet_manager.cc:1939] T 56a8c43c0533437997eea0a160ff01fe P 5b8b00ef016942a181ecac06da99bb39: Deleting consensus metadata
I20260621 02:38:25.450141 2768 ts_tablet_manager.cc:1918] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:38:25.450227 2770 ts_tablet_manager.cc:1939] T 56a8c43c0533437997eea0a160ff01fe P f68fddd4fd494c32ad9846e399193e15: Deleting consensus metadata
I20260621 02:38:25.450253 2768 log.cc:1196] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/56a8c43c0533437997eea0a160ff01fe
I20260621 02:38:25.450695 2768 ts_tablet_manager.cc:1939] T 56a8c43c0533437997eea0a160ff01fe P d2501fe77d7e49199da3ae1f5ba9cc88: Deleting consensus metadata
I20260621 02:38:25.451221 2119 catalog_manager.cc:4841] TS 5b8b00ef016942a181ecac06da99bb39 (127.0.97.3:35265): tablet 56a8c43c0533437997eea0a160ff01fe (table test-table [id=b0128441e3c540f291551cc9c7fef7bc]) successfully deleted
I20260621 02:38:25.451530 2119 catalog_manager.cc:4841] TS f68fddd4fd494c32ad9846e399193e15 (127.0.97.2:37587): tablet 56a8c43c0533437997eea0a160ff01fe (table test-table [id=b0128441e3c540f291551cc9c7fef7bc]) successfully deleted
I20260621 02:38:25.451602 2116 catalog_manager.cc:4841] TS d2501fe77d7e49199da3ae1f5ba9cc88 (127.0.97.1:34105): tablet 56a8c43c0533437997eea0a160ff01fe (table test-table [id=b0128441e3c540f291551cc9c7fef7bc]) successfully deleted
Jun 21 02:38:25 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:25.459409 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:25.432409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.0.97.1:34105)
0621 02:38:25.441106 (+ 8697us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:25.441115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:25.441147 (+ 32us) server_negotiation.cc:407] Connection header received
0621 02:38:25.442508 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:25.442528 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:25.442618 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:25.442707 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:25.444703 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:25.445893 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:25.447711 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:25.448458 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:25.448572 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:25.454517 (+ 5945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:25.454530 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:25.454545 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:25.454572 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:25.457122 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:25.457787 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:25.457794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:25.457796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:25.457921 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:25.458224 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:25.458227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:25.458229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:25.458517 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:25.458736 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:25.459000 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 02:38:25.459167 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8502,"thread_start_us":8370,"threads_started":1}
W20260621 02:38:25.460315 2334 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50616
W20260621 02:38:25.460846 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:38:25.461062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 2203
I20260621 02:38:25.486368 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2343
Jun 21 02:38:25 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:38:25.503168 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:25.487089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.0.97.2:37587)
0621 02:38:25.487616 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:25.487620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:25.487635 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:38:25.487686 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:25.487690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:25.487751 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:25.487840 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:25.488938 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:25.489697 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:25.490872 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:25.491240 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:25.491330 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:25.498065 (+ 6735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:25.498089 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:25.498104 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:25.498146 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:25.500794 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:25.501507 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:25.501514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:25.501516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:25.501664 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:25.501995 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:25.501998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:25.502000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:25.502326 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:25.502528 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:25.502762 (+ 234us) server_negotiation.cc:299] Negotiation successful
0621 02:38:25.502911 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":284,"threads_started":1}
W20260621 02:38:25.503823 2473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41920
W20260621 02:38:25.504441 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2343: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:38:25.504602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 2343
I20260621 02:38:25.526216 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2482
Jun 21 02:38:25 dist-test-slave-jkn7 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:38:25.540700 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:25.527006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.0.97.3:35265)
0621 02:38:25.527568 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:25.527573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:25.527588 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:38:25.527641 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:25.527644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:25.527712 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:25.527800 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:25.529068 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:25.529892 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:25.531114 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:25.531486 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:25.531589 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:25.535729 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:25.535744 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:25.535760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:25.535792 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:25.538528 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:25.539110 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:25.539117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:25.539119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:25.539261 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:25.539555 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:25.539558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:25.539560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:25.539866 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:25.540078 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:25.540279 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 02:38:25.540429 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
W20260621 02:38:25.541474 2612 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60730
W20260621 02:38:25.542084 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2482: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:38:25.542224 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 2482
I20260621 02:38:25.559854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2100
I20260621 02:38:25.560985 2172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:25.675498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 2100
2026-06-21T02:38:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8425 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:25 dist-test-slave-jkn7 krb5kdc[2778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:25 dist-test-slave-jkn7 krb5kdc[2778](info): set up 2 sockets
Jun 21 02:38:25 dist-test-slave-jkn7 krb5kdc[2778](info): commencing operation
krb5kdc: starting...
W20260621 02:38:26.757645 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:26 dist-test-slave-jkn7 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:26.936159 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35099
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:27.262313 2791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:27.262555 2791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:27.262722 2791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:27.269407 2791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:27.269492 2791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:27.269577 2791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:27.269685 2791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:27.269747 2791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:27.278788 2791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:27.279968 2791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:27.281245 2791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:27.292573 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:27.292662 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:27.292637 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:27.293493 2791 server_base.cc:1034] running on GCE node
I20260621 02:38:27.294183 2791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:27.295395 2791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:27.296602 2791 hybrid_clock.cc:648] HybridClock initialized: now 1782009507296570 us; error 47 us; skew 500 ppm
I20260621 02:38:27.296900 2791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:27 dist-test-slave-jkn7 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:27.299816 2791 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:27.301107 2791 webserver.cc:460] Webserver started at http://127.0.97.62:42019/ using document root <none> and password file <none>
I20260621 02:38:27.301780 2791 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:27.301920 2791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:27.302205 2791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:27.304759 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "dd71ef17efc74f05a93d895be9752f5a"
format_stamp: "Formatted at 2026-06-21 02:38:27 on dist-test-slave-jkn7"
server_key: "9a5ffc6c6897097338ae286cd7f74bac"
server_key_iv: "4861546744e8406dca7810eb9526d95f"
server_key_version: "encryptionkey@0"
I20260621 02:38:27.305554 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd71ef17efc74f05a93d895be9752f5a"
format_stamp: "Formatted at 2026-06-21 02:38:27 on dist-test-slave-jkn7"
server_key: "9a5ffc6c6897097338ae286cd7f74bac"
server_key_iv: "4861546744e8406dca7810eb9526d95f"
server_key_version: "encryptionkey@0"
I20260621 02:38:27.309902 2791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:38:27.313163 2806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:27.314460 2791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:27.314636 2791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "dd71ef17efc74f05a93d895be9752f5a"
format_stamp: "Formatted at 2026-06-21 02:38:27 on dist-test-slave-jkn7"
server_key: "9a5ffc6c6897097338ae286cd7f74bac"
server_key_iv: "4861546744e8406dca7810eb9526d95f"
server_key_version: "encryptionkey@0"
I20260621 02:38:27.314855 2791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:27.329645 2791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:27.334002 2791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:27.334299 2791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:27.352193 2791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35099
I20260621 02:38:27.352226 2857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35099 every 8 connection(s)
I20260621 02:38:27.353850 2791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:27.361651 2858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:27.362259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 2791
I20260621 02:38:27.362483 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:27.363173 388 external_mini_cluster.cc:1428] Setting key b075d64642bd235912840246fddd6186
I20260621 02:38:27.377600 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a: Bootstrap starting.
Jun 21 02:38:27 dist-test-slave-jkn7 krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:27.381425 2858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:27.382762 2858 log.cc:824] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:27.386003 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a: No bootstrap required, opened a new log
I20260621 02:38:27.387880 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:27.365772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.0.97.62:35099)
0621 02:38:27.366529 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:27.366549 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:27.366599 (+ 50us) server_negotiation.cc:407] Connection header received
0621 02:38:27.368631 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:27.368658 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:27.369172 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:27.369625 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:27.370890 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:27.372804 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:27.374160 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:27.375244 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:27.375436 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:27.379605 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:27.379616 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:27.379641 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:27.379690 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:27.383634 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:27.384373 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:27.384381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:27.384388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:27.384584 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:27.384911 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:27.384917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:27.384920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:27.385593 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:27.385880 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:27.386186 (+ 306us) server_negotiation.cc:299] Negotiation successful
0621 02:38:27.386614 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":317,"threads_started":1}
I20260621 02:38:27.394289 2858 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd71ef17efc74f05a93d895be9752f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35099 } }
I20260621 02:38:27.394759 2858 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:27.394888 2858 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd71ef17efc74f05a93d895be9752f5a, State: Initialized, Role: FOLLOWER
I20260621 02:38:27.395423 2858 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd71ef17efc74f05a93d895be9752f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35099 } }
I20260621 02:38:27.395673 2858 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:27.395807 2858 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:27.395979 2858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:27.398140 2858 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd71ef17efc74f05a93d895be9752f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35099 } }
I20260621 02:38:27.398640 2858 leader_election.cc:302] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd71ef17efc74f05a93d895be9752f5a; no voters:
I20260621 02:38:27.399211 2858 leader_election.cc:288] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:27.399350 2863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:27.400038 2863 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [term 1 LEADER]: Becoming Leader. State: Replica: dd71ef17efc74f05a93d895be9752f5a, State: Running, Role: LEADER
I20260621 02:38:27.400560 2863 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd71ef17efc74f05a93d895be9752f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35099 } }
I20260621 02:38:27.401448 2858 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:27.403600 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd71ef17efc74f05a93d895be9752f5a. Latest consensus state: current_term: 1 leader_uuid: "dd71ef17efc74f05a93d895be9752f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd71ef17efc74f05a93d895be9752f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35099 } } }
I20260621 02:38:27.403898 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:27.404043 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd71ef17efc74f05a93d895be9752f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd71ef17efc74f05a93d895be9752f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35099 } } }
I20260621 02:38:27.404350 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:27.406291 2869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:27.411275 2869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:27.420298 2869 catalog_manager.cc:1347] Generated new cluster ID: 7da56f7581864bcca5a3bad86f7a5d35
I20260621 02:38:27.420405 2869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:27.431192 2869 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:27.432453 2869 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:27.443408 2869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd71ef17efc74f05a93d895be9752f5a: Generated new TSK 0
I20260621 02:38:27.444177 2869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:38:27.494716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35099
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:27.827834 2884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:27.828102 2884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:27.828262 2884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:27.835227 2884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:27.835323 2884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:27.835544 2884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:27.844784 2884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:27.846006 2884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:27.847276 2884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:27.859977 2889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:27.860083 2890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:27.860010 2892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:27.860852 2884 server_base.cc:1034] running on GCE node
I20260621 02:38:27.861441 2884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:27.862365 2884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:27.863576 2884 hybrid_clock.cc:648] HybridClock initialized: now 1782009507863525 us; error 44 us; skew 500 ppm
I20260621 02:38:27.863853 2884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:27 dist-test-slave-jkn7 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:27.866918 2884 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:38:27.868335 2884 webserver.cc:460] Webserver started at http://127.0.97.1:34181/ using document root <none> and password file <none>
I20260621 02:38:27.868974 2884 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:27.869117 2884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:27.869478 2884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:27.872221 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "e53f7e58d76b423d84bbd2c409d5cded"
format_stamp: "Formatted at 2026-06-21 02:38:27 on dist-test-slave-jkn7"
server_key: "1af0008885da18f5265778ab9efd0401"
server_key_iv: "2af92eb718409947101be76deecb86cd"
server_key_version: "encryptionkey@0"
I20260621 02:38:27.873023 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e53f7e58d76b423d84bbd2c409d5cded"
format_stamp: "Formatted at 2026-06-21 02:38:27 on dist-test-slave-jkn7"
server_key: "1af0008885da18f5265778ab9efd0401"
server_key_iv: "2af92eb718409947101be76deecb86cd"
server_key_version: "encryptionkey@0"
I20260621 02:38:27.877830 2884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260621 02:38:27.881076 2899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:27.882478 2884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:27.882673 2884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "e53f7e58d76b423d84bbd2c409d5cded"
format_stamp: "Formatted at 2026-06-21 02:38:27 on dist-test-slave-jkn7"
server_key: "1af0008885da18f5265778ab9efd0401"
server_key_iv: "2af92eb718409947101be76deecb86cd"
server_key_version: "encryptionkey@0"
I20260621 02:38:27.882843 2884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:27.893637 2884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:27.897347 2884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:27.897620 2884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:27.898470 2884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:27.899686 2884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:27.899758 2884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:27.899916 2884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:27.899976 2884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:27.933145 2884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38653
I20260621 02:38:27.933193 3011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38653 every 8 connection(s)
I20260621 02:38:27.934474 2884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:27.943913 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 2884
I20260621 02:38:27.944139 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:27.944734 388 external_mini_cluster.cc:1428] Setting key 30da2aa2aff032df0c7d5281b4d72e2b
Jun 21 02:38:27 dist-test-slave-jkn7 krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:27.954998 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:27.937071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36417 (local address 127.0.97.62:35099)
0621 02:38:27.937560 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:27.937564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:27.938526 (+ 962us) server_negotiation.cc:407] Connection header received
0621 02:38:27.940366 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:27.940371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:27.940462 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:27.940586 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:27.942233 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:27.943146 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:27.944830 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:27.945421 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:27.945568 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:27.949261 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:27.949267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:27.949271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:27.949363 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:27.951855 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:27.952629 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:27.952633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:27.952636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:27.952751 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:27.953200 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:27.953206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:27.953210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:27.953545 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:27.953704 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:27.954415 (+ 711us) server_negotiation.cc:299] Negotiation successful
0621 02:38:27.954645 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":277,"threads_started":1}
I20260621 02:38:27.956034 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:27.937634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35099 (local address 127.0.97.1:36417)
0621 02:38:27.938349 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:27.938388 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:27.940063 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:27.940830 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:27.940838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:27.941448 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:27.942033 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:27.942050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:27.943327 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:27.943331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:27.944688 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:27.944699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:27.945653 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:27.945659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:27.945956 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:27.946764 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:27.946793 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:27.949003 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:27.952017 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:27.952026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:27.952046 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:27.952487 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:27.952862 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:27.952868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:27.952871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:27.953068 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:27.953684 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:27.953692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:27.954045 (+ 353us) client_negotiation.cc:769] Sending connection context
0621 02:38:27.954333 (+ 288us) client_negotiation.cc:240] Negotiation successful
0621 02:38:27.954893 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":284,"threads_started":1}
I20260621 02:38:27.957731 3012 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35099
I20260621 02:38:27.958057 3012 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:27.958770 3012 heartbeater.cc:511] Master 127.0.97.62:35099 requested a full tablet report, sending...
I20260621 02:38:27.961231 2823 ts_manager.cc:194] Registered new tserver with Master: e53f7e58d76b423d84bbd2c409d5cded (127.0.97.1:38653)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260621 02:38:27.963001 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:38:27.984551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35099
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:28.298622 3018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:28.298929 3018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:28.299091 3018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:28.306526 3018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:28.306643 3018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:28.306851 3018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:28.316555 3018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:28.317875 3018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:28.319278 3018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:28.332363 3023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:28.332422 3026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:28.332433 3024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:28.334589 3018 server_base.cc:1034] running on GCE node
I20260621 02:38:28.335186 3018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:28.336094 3018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:28.337384 3018 hybrid_clock.cc:648] HybridClock initialized: now 1782009508337334 us; error 40 us; skew 500 ppm
I20260621 02:38:28.337713 3018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:28 dist-test-slave-jkn7 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:28.340798 3018 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:38:28.342350 3018 webserver.cc:460] Webserver started at http://127.0.97.2:33469/ using document root <none> and password file <none>
I20260621 02:38:28.343047 3018 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:28.343180 3018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:28.343498 3018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:28.346383 3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "7db3e98082e04a989b657ab88dc97992"
format_stamp: "Formatted at 2026-06-21 02:38:28 on dist-test-slave-jkn7"
server_key: "beef3096c3fe8ca248bf601ab4f667c7"
server_key_iv: "b40375476db452392b3a67d0ad4d2354"
server_key_version: "encryptionkey@0"
I20260621 02:38:28.347141 3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7db3e98082e04a989b657ab88dc97992"
format_stamp: "Formatted at 2026-06-21 02:38:28 on dist-test-slave-jkn7"
server_key: "beef3096c3fe8ca248bf601ab4f667c7"
server_key_iv: "b40375476db452392b3a67d0ad4d2354"
server_key_version: "encryptionkey@0"
I20260621 02:38:28.352249 3018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:38:28.356474 3033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:28.358062 3018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:28.358238 3018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "7db3e98082e04a989b657ab88dc97992"
format_stamp: "Formatted at 2026-06-21 02:38:28 on dist-test-slave-jkn7"
server_key: "beef3096c3fe8ca248bf601ab4f667c7"
server_key_iv: "b40375476db452392b3a67d0ad4d2354"
server_key_version: "encryptionkey@0"
I20260621 02:38:28.358454 3018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:28.383970 3018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:28.388418 3018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:28.388679 3018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:28.389657 3018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:28.390934 3018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:28.391004 3018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:28.391085 3018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:28.391134 3018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:28.423566 3018 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34031
I20260621 02:38:28.423619 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34031 every 8 connection(s)
I20260621 02:38:28.424808 3018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:28.431411 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 3018
I20260621 02:38:28.431607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:28.432128 388 external_mini_cluster.cc:1428] Setting key 94c51abce9d4a68862954a309edc4ded
Jun 21 02:38:28 dist-test-slave-jkn7 krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:28.446359 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:28.427351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55785 (local address 127.0.97.62:35099)
0621 02:38:28.427536 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:28.427541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:28.428843 (+ 1302us) server_negotiation.cc:407] Connection header received
0621 02:38:28.430725 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:28.430731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:28.430799 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:28.430898 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:28.432950 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:28.433801 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:28.436229 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:28.436598 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:28.436695 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:28.440748 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:28.440753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:28.440756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:28.440786 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:28.443378 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:28.444101 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:28.444107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:28.444111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:28.444240 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:28.444660 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:28.444667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:28.444670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:28.444927 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:28.445081 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:28.445799 (+ 718us) server_negotiation.cc:299] Negotiation successful
0621 02:38:28.446022 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:38:28.447309 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:28.427869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35099 (local address 127.0.97.2:55785)
0621 02:38:28.428682 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:28.428727 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:28.430452 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:28.431244 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:28.431258 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:28.431994 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:28.432767 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:28.432788 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:28.433976 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:28.433980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:28.436091 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:28.436109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:28.436829 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:28.436838 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:28.437191 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:28.437929 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:28.437962 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:28.440520 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:28.443551 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:28.443560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:28.443577 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:28.443977 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:28.444358 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:28.444363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:28.444367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:28.444546 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:28.445060 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:28.445068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:28.445458 (+ 390us) client_negotiation.cc:769] Sending connection context
0621 02:38:28.445729 (+ 271us) client_negotiation.cc:240] Negotiation successful
0621 02:38:28.446253 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":356,"threads_started":1}
I20260621 02:38:28.448757 3146 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35099
I20260621 02:38:28.449054 3146 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:28.449795 3146 heartbeater.cc:511] Master 127.0.97.62:35099 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:38:28.451282 2823 ts_manager.cc:194] Registered new tserver with Master: 7db3e98082e04a989b657ab88dc97992 (127.0.97.2:34031)
I20260621 02:38:28.452107 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:38:28.474584 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35099
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:28.793957 3151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:28.794224 3151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:28.794375 3151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:28.801622 3151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:28.801725 3151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:28.801884 3151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:28.811867 3151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:28.813241 3151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:28.814664 3151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:28.827632 3156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:28.827706 3159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:28.827680 3157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:28.828545 3151 server_base.cc:1034] running on GCE node
I20260621 02:38:28.829185 3151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:28.830207 3151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:28.831420 3151 hybrid_clock.cc:648] HybridClock initialized: now 1782009508831388 us; error 47 us; skew 500 ppm
I20260621 02:38:28.831692 3151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:28 dist-test-slave-jkn7 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:28.834808 3151 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:38:28.836287 3151 webserver.cc:460] Webserver started at http://127.0.97.3:45287/ using document root <none> and password file <none>
I20260621 02:38:28.836961 3151 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:28.837090 3151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:28.837478 3151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:28.840323 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "bc8ca0d8b13d4a5c948e6080da6abc3f"
format_stamp: "Formatted at 2026-06-21 02:38:28 on dist-test-slave-jkn7"
server_key: "f83b16f738c48d9cf84c5a28e381df26"
server_key_iv: "6b37bbc92d124a36e24984e626feb696"
server_key_version: "encryptionkey@0"
I20260621 02:38:28.841213 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bc8ca0d8b13d4a5c948e6080da6abc3f"
format_stamp: "Formatted at 2026-06-21 02:38:28 on dist-test-slave-jkn7"
server_key: "f83b16f738c48d9cf84c5a28e381df26"
server_key_iv: "6b37bbc92d124a36e24984e626feb696"
server_key_version: "encryptionkey@0"
I20260621 02:38:28.846419 3151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:38:28.850473 3166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:28.851845 3151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:38:28.851992 3151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "bc8ca0d8b13d4a5c948e6080da6abc3f"
format_stamp: "Formatted at 2026-06-21 02:38:28 on dist-test-slave-jkn7"
server_key: "f83b16f738c48d9cf84c5a28e381df26"
server_key_iv: "6b37bbc92d124a36e24984e626feb696"
server_key_version: "encryptionkey@0"
I20260621 02:38:28.852167 3151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:28.879474 3151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:28.883427 3151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:28.883657 3151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:28.884493 3151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:28.885833 3151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:28.885903 3151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:28.885982 3151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:28.886026 3151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:28.918220 3151 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32831
I20260621 02:38:28.918249 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32831 every 8 connection(s)
I20260621 02:38:28.919652 3151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:28.921940 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 3151
I20260621 02:38:28.922175 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:28.922610 388 external_mini_cluster.cc:1428] Setting key d2113cdd12eea7b6d2667002c9abf50c
Jun 21 02:38:28 dist-test-slave-jkn7 krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:28.939929 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:28.922239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46029 (local address 127.0.97.62:35099)
0621 02:38:28.922455 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:28.922462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:28.923812 (+ 1350us) server_negotiation.cc:407] Connection header received
0621 02:38:28.925739 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:28.925745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:28.925833 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:28.925937 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:28.927750 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:28.928477 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:28.930046 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:28.930465 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:28.930602 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:28.933904 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:28.933911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:28.933916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:28.933952 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:28.936822 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:28.937763 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:28.937768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:28.937771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:28.937929 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:28.938392 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:28.938396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:28.938398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:28.938745 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:28.938854 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:28.939532 (+ 678us) server_negotiation.cc:299] Negotiation successful
0621 02:38:28.939683 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 02:38:28.940714 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:28.922866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35099 (local address 127.0.97.3:46029)
0621 02:38:28.923654 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:28.923704 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:28.925477 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:28.926204 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:28.926214 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:28.926957 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:28.927583 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:28.927616 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:28.928625 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:28.928631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:28.929905 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:28.929920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:28.930683 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:28.930691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:28.930924 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:28.931574 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:28.931596 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:28.933703 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:28.936958 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:28.936968 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:28.936984 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:28.937580 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:28.938043 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:28.938050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:28.938053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:28.938270 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:28.938865 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:28.938873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:28.939150 (+ 277us) client_negotiation.cc:769] Sending connection context
0621 02:38:28.939370 (+ 220us) client_negotiation.cc:240] Negotiation successful
0621 02:38:28.939700 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":348,"threads_started":1}
I20260621 02:38:28.942011 3279 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35099
I20260621 02:38:28.942301 3279 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:28.942939 3279 heartbeater.cc:511] Master 127.0.97.62:35099 requested a full tablet report, sending...
I20260621 02:38:28.944222 2823 ts_manager.cc:194] Registered new tserver with Master: bc8ca0d8b13d4a5c948e6080da6abc3f (127.0.97.3:32831)
I20260621 02:38:28.944941 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46029
I20260621 02:38:28.950780 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:38:28.965853 3012 heartbeater.cc:503] Master 127.0.97.62:35099 was elected leader, sending a full tablet report...
W20260621 02:38:28.978343 3014 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:38:28.976138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.0.97.62:35099)
0621 02:38:28.976311 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:28.976316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:28.976754 (+ 438us) server_negotiation.cc:407] Connection header received
0621 02:38:28.976887 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:28.976890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:28.976945 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:28.977030 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:28.978161 (+ 1131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44930: BlockingRecv error: recv got EOF from 127.0.0.1:44930 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:38:28.981592 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2884
W20260621 02:38:28.984122 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2884: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:38653: server requires authentication, but client does not have Kerberos credentials available
I20260621 02:38:28.984299 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 2884
I20260621 02:38:28.999943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3018
W20260621 02:38:29.002650 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3018: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34031: server requires authentication, but client does not have Kerberos credentials available
I20260621 02:38:29.002812 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 3018
I20260621 02:38:29.017486 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3151
W20260621 02:38:29.020089 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3151: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:32831: server requires authentication, but client does not have Kerberos credentials available
I20260621 02:38:29.020237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 3151
I20260621 02:38:29.034911 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 2791
I20260621 02:38:29.035919 2853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:29.135054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 2791
2026-06-21T02:38:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3456 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:29 dist-test-slave-jkn7 krb5kdc[3296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:29 dist-test-slave-jkn7 krb5kdc[3296](info): set up 2 sockets
Jun 21 02:38:29 dist-test-slave-jkn7 krb5kdc[3296](info): commencing operation
krb5kdc: starting...
W20260621 02:38:30.213585 388 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:30 dist-test-slave-jkn7 krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:30.388504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44965
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:30.696727 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:30.697000 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:30.697187 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:30.704250 3309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:30.704342 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:30.704442 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:30.704541 3309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:30.704627 3309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:30.714112 3309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:30.715322 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:30.716629 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:30.728286 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:30.728367 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:30.728346 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:30.729100 3309 server_base.cc:1034] running on GCE node
I20260621 02:38:30.729918 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:30.731278 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:30.732519 3309 hybrid_clock.cc:648] HybridClock initialized: now 1782009510732475 us; error 43 us; skew 500 ppm
I20260621 02:38:30.732915 3309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:30 dist-test-slave-jkn7 krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:30.740108 3309 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:30.741787 3309 webserver.cc:460] Webserver started at http://127.0.97.62:32957/ using document root <none> and password file <none>
I20260621 02:38:30.742435 3309 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:30.742568 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:30.742861 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:30.746209 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "f49e604f2d094ddc8ef3f243c4a673bd"
format_stamp: "Formatted at 2026-06-21 02:38:30 on dist-test-slave-jkn7"
server_key: "3e825393ee3b26cb8c6c7fd2057e5ae1"
server_key_iv: "1c53399c14564981c6b6e657ac41b987"
server_key_version: "encryptionkey@0"
I20260621 02:38:30.746942 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "f49e604f2d094ddc8ef3f243c4a673bd"
format_stamp: "Formatted at 2026-06-21 02:38:30 on dist-test-slave-jkn7"
server_key: "3e825393ee3b26cb8c6c7fd2057e5ae1"
server_key_iv: "1c53399c14564981c6b6e657ac41b987"
server_key_version: "encryptionkey@0"
I20260621 02:38:30.751919 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:38:30.755293 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:30.756697 3309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 02:38:30.756902 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "f49e604f2d094ddc8ef3f243c4a673bd"
format_stamp: "Formatted at 2026-06-21 02:38:30 on dist-test-slave-jkn7"
server_key: "3e825393ee3b26cb8c6c7fd2057e5ae1"
server_key_iv: "1c53399c14564981c6b6e657ac41b987"
server_key_version: "encryptionkey@0"
I20260621 02:38:30.757066 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:30.769424 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:30.773350 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:30.773617 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:30.791231 3309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44965
I20260621 02:38:30.791268 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44965 every 8 connection(s)
I20260621 02:38:30.792493 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:30.793843 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 3309
I20260621 02:38:30.794049 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:30.794831 388 external_mini_cluster.cc:1428] Setting key 14a879b9c4110ce1a64655f82f5470cb
I20260621 02:38:30.797019 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:38:30 dist-test-slave-jkn7 krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:30.809696 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd: Bootstrap starting.
I20260621 02:38:30.812716 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:30.813750 3376 log.cc:824] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:30.816413 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:30.796617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.0.97.62:44965)
0621 02:38:30.797458 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:30.797470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:30.797508 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:38:30.799007 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:30.799026 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:30.799381 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:30.799682 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:30.801473 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:30.802752 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:30.804438 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:30.805039 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:30.805154 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:30.809581 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:30.809589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:30.809601 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:30.809636 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:30.812534 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:30.813272 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:30.813304 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:30.813315 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:30.813473 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:30.813843 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:30.813850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:30.813853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:30.814543 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:30.814834 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:30.815123 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 02:38:30.815489 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":358,"threads_started":1}
I20260621 02:38:30.816756 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd: No bootstrap required, opened a new log
I20260621 02:38:30.820119 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:38:30.824025 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f49e604f2d094ddc8ef3f243c4a673bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260621 02:38:30.824342 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:30.824460 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49e604f2d094ddc8ef3f243c4a673bd, State: Initialized, Role: FOLLOWER
I20260621 02:38:30.825016 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f49e604f2d094ddc8ef3f243c4a673bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260621 02:38:30.825235 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:30.825383 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:30.825556 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:30.828994 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f49e604f2d094ddc8ef3f243c4a673bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260621 02:38:30.829550 3376 leader_election.cc:302] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f49e604f2d094ddc8ef3f243c4a673bd; no voters:
I20260621 02:38:30.830184 3376 leader_election.cc:288] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:30.830291 3382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:30.831785 3382 raft_consensus.cc:686] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [term 1 LEADER]: Becoming Leader. State: Replica: f49e604f2d094ddc8ef3f243c4a673bd, State: Running, Role: LEADER
I20260621 02:38:30.832296 3382 consensus_queue.cc:237] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f49e604f2d094ddc8ef3f243c4a673bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260621 02:38:30.832495 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:30.834895 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f49e604f2d094ddc8ef3f243c4a673bd. Latest consensus state: current_term: 1 leader_uuid: "f49e604f2d094ddc8ef3f243c4a673bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f49e604f2d094ddc8ef3f243c4a673bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } } }
I20260621 02:38:30.835265 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:30.835764 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f49e604f2d094ddc8ef3f243c4a673bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f49e604f2d094ddc8ef3f243c4a673bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } } }
I20260621 02:38:30.836048 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:30.837857 3389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:30.844926 3389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:30.853826 3389 catalog_manager.cc:1347] Generated new cluster ID: b5f42be9d85c461ea11a9cb038de7f4f
I20260621 02:38:30.853928 3389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:30.864068 3389 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:30.865779 3389 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:30.867139 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:30.858611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.0.97.62:44965)
0621 02:38:30.858790 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:30.858798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:30.859590 (+ 792us) server_negotiation.cc:407] Connection header received
0621 02:38:30.859838 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:30.859842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:30.859929 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:30.860054 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:30.861218 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:30.862088 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:30.863958 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:30.864440 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:30.864593 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:30.865659 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:30.865667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:30.865676 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:30.866353 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:30.866420 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:30.866448 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:30.866572 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:30.866697 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 02:38:30.866819 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:38:30.872696 3389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f49e604f2d094ddc8ef3f243c4a673bd: Generated new TSK 0
I20260621 02:38:30.873636 3389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:38:30.886603 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3309
I20260621 02:38:30.887844 3371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:30.973734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 3309
2026-06-21T02:38:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1833 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:31 dist-test-slave-jkn7 krb5kdc[3410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:31 dist-test-slave-jkn7 krb5kdc[3410](info): set up 2 sockets
Jun 21 02:38:31 dist-test-slave-jkn7 krb5kdc[3410](info): commencing operation
krb5kdc: starting...
W20260621 02:38:32.053494 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.010s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:32 dist-test-slave-jkn7 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:32.230157 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42865
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:32.530879 3423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:32.531150 3423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:32.531360 3423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:32.538255 3423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:32.538342 3423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:32.538425 3423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:32.538529 3423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:32.538620 3423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:32.547741 3423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:32.548946 3423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:32.550213 3423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:32.561434 3429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:32.561517 3428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:32.561491 3431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:32.562211 3423 server_base.cc:1034] running on GCE node
I20260621 02:38:32.562934 3423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:32.564291 3423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:32.565521 3423 hybrid_clock.cc:648] HybridClock initialized: now 1782009512565481 us; error 54 us; skew 500 ppm
I20260621 02:38:32.565869 3423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:32 dist-test-slave-jkn7 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:32.568941 3423 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:32.570461 3423 webserver.cc:460] Webserver started at http://127.0.97.62:34199/ using document root <none> and password file <none>
I20260621 02:38:32.571118 3423 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:32.571269 3423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:32.571563 3423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:32.574447 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "5829700a22194105b9e16d9a8cd6b1fe"
format_stamp: "Formatted at 2026-06-21 02:38:32 on dist-test-slave-jkn7"
server_key: "15ccf526a28e6bcd0b1a6a8ea3339fb1"
server_key_iv: "d9b8d8794c9909d0f20fdfd493fe49b6"
server_key_version: "encryptionkey@0"
I20260621 02:38:32.575179 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "5829700a22194105b9e16d9a8cd6b1fe"
format_stamp: "Formatted at 2026-06-21 02:38:32 on dist-test-slave-jkn7"
server_key: "15ccf526a28e6bcd0b1a6a8ea3339fb1"
server_key_iv: "d9b8d8794c9909d0f20fdfd493fe49b6"
server_key_version: "encryptionkey@0"
I20260621 02:38:32.579977 3423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260621 02:38:32.583173 3438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:32.584424 3423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:38:32.584596 3423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "5829700a22194105b9e16d9a8cd6b1fe"
format_stamp: "Formatted at 2026-06-21 02:38:32 on dist-test-slave-jkn7"
server_key: "15ccf526a28e6bcd0b1a6a8ea3339fb1"
server_key_iv: "d9b8d8794c9909d0f20fdfd493fe49b6"
server_key_version: "encryptionkey@0"
I20260621 02:38:32.584776 3423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:32.597936 3423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:32.602028 3423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:32.602314 3423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:32.620723 3423 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42865
I20260621 02:38:32.620754 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42865 every 8 connection(s)
I20260621 02:38:32.622124 3423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:32.625003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 3423
I20260621 02:38:32.625214 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:32.626060 388 external_mini_cluster.cc:1428] Setting key 3fe6df0c88a441e7213040a48919b59b
I20260621 02:38:32.626251 3490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:32.640463 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe: Bootstrap starting.
Jun 21 02:38:32 dist-test-slave-jkn7 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:32.644217 3490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:32.645470 3490 log.cc:824] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:32.648737 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe: No bootstrap required, opened a new log
I20260621 02:38:32.650063 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:32.628032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.0.97.62:42865)
0621 02:38:32.628783 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:32.628795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:32.628837 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:38:32.631209 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:32.631231 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:32.631595 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:32.631915 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:32.633563 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:32.634810 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:32.636490 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:32.637166 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:32.637350 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:32.641834 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:32.641846 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:32.641869 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:32.641921 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:32.645765 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:32.646602 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:32.646608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:32.646613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:32.646758 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:32.647126 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:32.647131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:32.647134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:32.647778 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:32.648072 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:32.648385 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 02:38:32.648788 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":349,"threads_started":1}
I20260621 02:38:32.656124 3490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829700a22194105b9e16d9a8cd6b1fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42865 } }
I20260621 02:38:32.656423 3490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:32.656493 3490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5829700a22194105b9e16d9a8cd6b1fe, State: Initialized, Role: FOLLOWER
I20260621 02:38:32.656914 3490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829700a22194105b9e16d9a8cd6b1fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42865 } }
I20260621 02:38:32.657117 3490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:32.657266 3490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:32.657475 3490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:32.659584 3490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829700a22194105b9e16d9a8cd6b1fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42865 } }
I20260621 02:38:32.660039 3490 leader_election.cc:302] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5829700a22194105b9e16d9a8cd6b1fe; no voters:
I20260621 02:38:32.660559 3490 leader_election.cc:288] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:32.660704 3495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:32.661458 3495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [term 1 LEADER]: Becoming Leader. State: Replica: 5829700a22194105b9e16d9a8cd6b1fe, State: Running, Role: LEADER
I20260621 02:38:32.661966 3495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829700a22194105b9e16d9a8cd6b1fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42865 } }
I20260621 02:38:32.662570 3490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:32.665266 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5829700a22194105b9e16d9a8cd6b1fe. Latest consensus state: current_term: 1 leader_uuid: "5829700a22194105b9e16d9a8cd6b1fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829700a22194105b9e16d9a8cd6b1fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42865 } } }
I20260621 02:38:32.665331 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5829700a22194105b9e16d9a8cd6b1fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829700a22194105b9e16d9a8cd6b1fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42865 } } }
I20260621 02:38:32.665788 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:32.665849 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:32.669950 3503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:32.675436 3503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:32.684307 3503 catalog_manager.cc:1347] Generated new cluster ID: 760605804bee4a1293729535998198a0
I20260621 02:38:32.684463 3503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:32.723179 3503 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:32.724475 3503 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:32.732091 3503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5829700a22194105b9e16d9a8cd6b1fe: Generated new TSK 0
I20260621 02:38:32.732872 3503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:38:32.790019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42865
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:33.108203 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:33.108471 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:33.108593 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:33.115468 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:33.115561 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:33.115665 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:33.124779 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:33.125969 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:33.127151 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:33.139837 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:33.139868 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:33.139832 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:33.145958 3516 server_base.cc:1034] running on GCE node
I20260621 02:38:33.146560 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:33.147516 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:33.148777 3516 hybrid_clock.cc:648] HybridClock initialized: now 1782009513148716 us; error 42 us; skew 500 ppm
I20260621 02:38:33.149178 3516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:33 dist-test-slave-jkn7 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:33.152577 3516 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:38:33.154033 3516 webserver.cc:460] Webserver started at http://127.0.97.1:36073/ using document root <none> and password file <none>
I20260621 02:38:33.154677 3516 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:33.154771 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:33.155001 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:33.157776 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "d972a79fc3904f339dc6c18b83b91264"
format_stamp: "Formatted at 2026-06-21 02:38:33 on dist-test-slave-jkn7"
server_key: "9aee7798e8d0a58853687e3c116d452a"
server_key_iv: "6d718ac3b891ec7af428c8238462ca19"
server_key_version: "encryptionkey@0"
I20260621 02:38:33.158483 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d972a79fc3904f339dc6c18b83b91264"
format_stamp: "Formatted at 2026-06-21 02:38:33 on dist-test-slave-jkn7"
server_key: "9aee7798e8d0a58853687e3c116d452a"
server_key_iv: "6d718ac3b891ec7af428c8238462ca19"
server_key_version: "encryptionkey@0"
I20260621 02:38:33.163551 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:38:33.167053 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:33.168416 3516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:33.168542 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "d972a79fc3904f339dc6c18b83b91264"
format_stamp: "Formatted at 2026-06-21 02:38:33 on dist-test-slave-jkn7"
server_key: "9aee7798e8d0a58853687e3c116d452a"
server_key_iv: "6d718ac3b891ec7af428c8238462ca19"
server_key_version: "encryptionkey@0"
I20260621 02:38:33.168704 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:33.204903 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:33.209400 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:33.209673 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:33.210574 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:33.211836 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:33.211910 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:33.211999 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:33.212042 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:33.244598 3516 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46019
I20260621 02:38:33.244696 3643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46019 every 8 connection(s)
I20260621 02:38:33.245949 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:33.250443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 3516
I20260621 02:38:33.250654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:33.251289 388 external_mini_cluster.cc:1428] Setting key b0c45db2c2fa8fa2794254163b476f00
Jun 21 02:38:33 dist-test-slave-jkn7 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:33.267705 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:33.248497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42333 (local address 127.0.97.62:42865)
0621 02:38:33.248953 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:33.248961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:33.249901 (+ 940us) server_negotiation.cc:407] Connection header received
0621 02:38:33.251832 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:33.251839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:33.251916 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:33.252011 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:33.253696 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:33.254874 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:33.256469 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:33.257201 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:33.257372 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:33.261266 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:33.261271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:33.261275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:33.261361 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:33.264399 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:33.265116 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:33.265122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:33.265127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:33.265291 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:33.265880 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:33.265884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:33.265886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:33.266157 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:33.266283 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:33.267277 (+ 994us) server_negotiation.cc:299] Negotiation successful
0621 02:38:33.267446 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":211,"threads_started":1}
I20260621 02:38:33.269271 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:33.249046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42865 (local address 127.0.97.1:42333)
0621 02:38:33.249753 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:33.249792 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:33.251545 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:33.252282 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:33.252291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:33.252779 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:33.253528 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:33.253547 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:33.255065 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:33.255069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:33.256316 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:33.256327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:33.257463 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:33.257468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:33.257666 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:33.258374 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:33.258417 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:33.261000 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:33.264542 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:33.264551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:33.264569 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:33.264986 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:33.265449 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:33.265456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:33.265463 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:33.265719 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:33.266300 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:33.266311 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:33.266803 (+ 492us) client_negotiation.cc:769] Sending connection context
0621 02:38:33.267161 (+ 358us) client_negotiation.cc:240] Negotiation successful
0621 02:38:33.267748 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":311,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260621 02:38:33.271075 3644 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42865
I20260621 02:38:33.271368 3644 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:33.272311 3644 heartbeater.cc:511] Master 127.0.97.62:42865 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260621 02:38:33.274466 3455 ts_manager.cc:194] Registered new tserver with Master: d972a79fc3904f339dc6c18b83b91264 (127.0.97.1:46019)
I20260621 02:38:33.276351 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:38:33.296191 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42865
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:33.609944 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:33.610260 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:33.610425 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:33.617342 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:33.617487 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:33.617651 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:33.631316 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:33.632879 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:33.634557 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:33.648298 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:33.648382 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:33.648377 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:33.649577 3650 server_base.cc:1034] running on GCE node
I20260621 02:38:33.650198 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:33.651098 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:33.652318 3650 hybrid_clock.cc:648] HybridClock initialized: now 1782009513652256 us; error 57 us; skew 500 ppm
I20260621 02:38:33.652647 3650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:33 dist-test-slave-jkn7 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:33.655680 3650 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:38:33.656975 3650 webserver.cc:460] Webserver started at http://127.0.97.2:36349/ using document root <none> and password file <none>
I20260621 02:38:33.657693 3650 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:33.657836 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:33.658116 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:33.660881 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "ccf6b0eff0b340d38ade321e44aa0ba7"
format_stamp: "Formatted at 2026-06-21 02:38:33 on dist-test-slave-jkn7"
server_key: "d57f2f71d70a9558b418c522dcfc5c2e"
server_key_iv: "1570a4dd9819ce7caeeb685f29d871fe"
server_key_version: "encryptionkey@0"
I20260621 02:38:33.662005 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ccf6b0eff0b340d38ade321e44aa0ba7"
format_stamp: "Formatted at 2026-06-21 02:38:33 on dist-test-slave-jkn7"
server_key: "d57f2f71d70a9558b418c522dcfc5c2e"
server_key_iv: "1570a4dd9819ce7caeeb685f29d871fe"
server_key_version: "encryptionkey@0"
I20260621 02:38:33.667169 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:38:33.670471 3665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:33.671898 3650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:38:33.672073 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "ccf6b0eff0b340d38ade321e44aa0ba7"
format_stamp: "Formatted at 2026-06-21 02:38:33 on dist-test-slave-jkn7"
server_key: "d57f2f71d70a9558b418c522dcfc5c2e"
server_key_iv: "1570a4dd9819ce7caeeb685f29d871fe"
server_key_version: "encryptionkey@0"
I20260621 02:38:33.672264 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:33.688196 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:33.692545 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:33.692837 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:33.693786 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:33.695034 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:33.695133 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:33.695298 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:33.695365 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:33.728107 3650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43907
I20260621 02:38:33.728152 3777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43907 every 8 connection(s)
I20260621 02:38:33.729373 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:33.732851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 3650
I20260621 02:38:33.733027 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:33.733635 388 external_mini_cluster.cc:1428] Setting key ff55055bfd20bf729e32ef08f6d67604
Jun 21 02:38:33 dist-test-slave-jkn7 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:33.749903 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:33.731864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39163 (local address 127.0.97.62:42865)
0621 02:38:33.732041 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:33.732046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:33.733397 (+ 1351us) server_negotiation.cc:407] Connection header received
0621 02:38:33.735477 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:33.735482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:33.735577 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:33.735682 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:33.737104 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:33.737847 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:33.739476 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:33.740114 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:33.740253 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:33.744063 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:33.744068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:33.744071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:33.744100 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:33.746645 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:33.747601 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:33.747605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:33.747607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:33.747741 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:33.748268 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:33.748274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:33.748276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:33.748527 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:33.748641 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:33.749505 (+ 864us) server_negotiation.cc:299] Negotiation successful
0621 02:38:33.749659 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:38:33.751446 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:33.732321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42865 (local address 127.0.97.2:39163)
0621 02:38:33.733187 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:33.733248 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:33.735226 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:33.735916 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:33.735924 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:33.736401 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:33.736945 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:33.736963 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:33.737998 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:33.738003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:33.739332 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:33.739344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:33.740322 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:33.740327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:33.740525 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:33.741208 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:33.741229 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:33.743855 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:33.746810 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:33.746821 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:33.746841 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:33.747466 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:33.747853 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:33.747861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:33.747867 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:33.748121 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:33.748657 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:33.748665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:33.749099 (+ 434us) client_negotiation.cc:769] Sending connection context
0621 02:38:33.749443 (+ 344us) client_negotiation.cc:240] Negotiation successful
0621 02:38:33.749994 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":308,"threads_started":1}
I20260621 02:38:33.752864 3778 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42865
I20260621 02:38:33.753118 3778 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260621 02:38:33.753885 3778 heartbeater.cc:511] Master 127.0.97.62:42865 requested a full tablet report, sending...
I20260621 02:38:33.755328 3455 ts_manager.cc:194] Registered new tserver with Master: ccf6b0eff0b340d38ade321e44aa0ba7 (127.0.97.2:43907)
I20260621 02:38:33.756103 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:38:33.778504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42865
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:34.107021 3783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:34.107328 3783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:34.107488 3783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:34.114660 3783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:34.114804 3783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:34.114959 3783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:34.124389 3783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:34.125702 3783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:34.127018 3783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:34.139835 3788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:34.139902 3791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:34.139878 3789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:34.140676 3783 server_base.cc:1034] running on GCE node
I20260621 02:38:34.141250 3783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:34.142238 3783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:34.143466 3783 hybrid_clock.cc:648] HybridClock initialized: now 1782009514143398 us; error 60 us; skew 500 ppm
I20260621 02:38:34.143764 3783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:34.147059 3783 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:38:34.148447 3783 webserver.cc:460] Webserver started at http://127.0.97.3:45243/ using document root <none> and password file <none>
I20260621 02:38:34.149147 3783 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:34.149277 3783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:34.149627 3783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:34.152447 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "6271c8cdd5964353a818348b1f8209bc"
format_stamp: "Formatted at 2026-06-21 02:38:34 on dist-test-slave-jkn7"
server_key: "bd6b7338b380a3e66be41c2758166e75"
server_key_iv: "7b8e05f2b09f1075d492205d94c4c98f"
server_key_version: "encryptionkey@0"
I20260621 02:38:34.153224 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6271c8cdd5964353a818348b1f8209bc"
format_stamp: "Formatted at 2026-06-21 02:38:34 on dist-test-slave-jkn7"
server_key: "bd6b7338b380a3e66be41c2758166e75"
server_key_iv: "7b8e05f2b09f1075d492205d94c4c98f"
server_key_version: "encryptionkey@0"
I20260621 02:38:34.158375 3783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:38:34.161593 3798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:34.162978 3783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:38:34.163105 3783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "6271c8cdd5964353a818348b1f8209bc"
format_stamp: "Formatted at 2026-06-21 02:38:34 on dist-test-slave-jkn7"
server_key: "bd6b7338b380a3e66be41c2758166e75"
server_key_iv: "7b8e05f2b09f1075d492205d94c4c98f"
server_key_version: "encryptionkey@0"
I20260621 02:38:34.163272 3783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:34.191268 3783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:34.195062 3783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:34.195310 3783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:34.196152 3783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:34.197443 3783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:34.197515 3783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:34.197595 3783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:34.197638 3783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:34.229647 3783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42449
I20260621 02:38:34.229704 3910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42449 every 8 connection(s)
I20260621 02:38:34.230919 3783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:34.235872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 3783
I20260621 02:38:34.236073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:34.236521 388 external_mini_cluster.cc:1428] Setting key 9741591299aa89cc41ce360d723c445f
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:34.252888 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:34.233520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41141 (local address 127.0.97.62:42865)
0621 02:38:34.233704 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:34.233708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:34.234916 (+ 1208us) server_negotiation.cc:407] Connection header received
0621 02:38:34.236885 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:34.236891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:34.237030 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:34.237149 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:34.239253 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.240251 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.242018 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.242435 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.242545 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:34.246331 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:34.246337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:34.246342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:34.246383 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:34.249630 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.250495 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.250501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.250505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.250644 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.251132 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.251136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.251139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.251496 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:34.251644 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:34.252492 (+ 848us) server_negotiation.cc:299] Negotiation successful
0621 02:38:34.252654 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:38:34.253881 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:34.234015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42865 (local address 127.0.97.3:41141)
0621 02:38:34.234741 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:34.234785 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:34.236552 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:34.237476 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:34.237490 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:34.238336 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:34.239073 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:34.239093 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.240421 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.240425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:34.241842 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:34.241854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.242646 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.242651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:34.242872 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:34.243622 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:34.243647 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:34.246093 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:34.249766 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:34.249774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:34.249788 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:34.250358 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:34.250766 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:34.250773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:34.250777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:34.251005 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:34.251617 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:34.251623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:34.252092 (+ 469us) client_negotiation.cc:769] Sending connection context
0621 02:38:34.252374 (+ 282us) client_negotiation.cc:240] Negotiation successful
0621 02:38:34.252760 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":277,"threads_started":1}
I20260621 02:38:34.255275 3911 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42865
I20260621 02:38:34.255589 3911 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:34.256304 3911 heartbeater.cc:511] Master 127.0.97.62:42865 requested a full tablet report, sending...
I20260621 02:38:34.257741 3455 ts_manager.cc:194] Registered new tserver with Master: 6271c8cdd5964353a818348b1f8209bc (127.0.97.3:42449)
I20260621 02:38:34.258494 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41141
I20260621 02:38:34.264652 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:38:34.279467 3644 heartbeater.cc:503] Master 127.0.97.62:42865 was elected leader, sending a full tablet report...
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009514, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009514, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:34.311340 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:34.297677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.0.97.62:42865)
0621 02:38:34.297845 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:34.297850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:34.298385 (+ 535us) server_negotiation.cc:407] Connection header received
0621 02:38:34.298534 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:34.298538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:34.298601 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:34.298694 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:34.299970 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.300713 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.302036 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.302435 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.302533 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:34.306322 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:34.306327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:34.306331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:34.306363 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:34.308990 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.309700 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.309704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.309706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.309821 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.310143 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.310146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.310148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.310468 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:34.310648 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:34.310932 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 02:38:34.311095 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260621 02:38:34.311919 3455 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37604
I20260621 02:38:34.313580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3516
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009514, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:34.329175 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:34.314605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.97.1:46019)
0621 02:38:34.315217 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:34.315224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:34.315248 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:38:34.315302 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:34.315308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:34.315526 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:34.315652 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:34.316707 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.317827 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.319021 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.319585 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.319683 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:34.323620 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:34.323633 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:34.323649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:34.323684 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:34.326725 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.327370 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.327379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.327381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.327506 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.327843 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.327846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.327848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.328174 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:34.328421 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:34.328693 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 02:38:34.328900 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":317,"threads_started":1}
W20260621 02:38:34.330507 3639 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59452
W20260621 02:38:34.331643 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3516: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:38:34.331851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 3516
I20260621 02:38:34.347432 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3650
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009514, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:38:34.362323 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:34.348168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.0.97.2:43907)
0621 02:38:34.348686 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:34.348693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:34.348718 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:38:34.348772 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:34.348780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:34.348991 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:34.349115 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:34.350341 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.351301 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.352507 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.353012 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.353109 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:34.356943 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:34.356954 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:34.356966 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:34.356995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:34.359856 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.360469 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.360477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.360479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.360599 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.360940 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.360944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.360946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.361364 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:34.361590 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:34.361849 (+ 259us) server_negotiation.cc:299] Negotiation successful
0621 02:38:34.362034 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":271,"threads_started":1}
W20260621 02:38:34.363519 3773 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46224
W20260621 02:38:34.364514 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3650: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:38:34.364663 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 3650
I20260621 02:38:34.379734 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3783
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009514, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:38:34.395283 3924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:34.380627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.0.97.3:42449)
0621 02:38:34.381257 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:34.381264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:34.381346 (+ 82us) server_negotiation.cc:407] Connection header received
0621 02:38:34.381405 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:34.381412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:34.381608 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:34.381772 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:34.383041 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.384013 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.385251 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:34.385857 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:34.385994 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:34.390103 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:34.390113 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:34.390126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:34.390155 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:34.392614 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.393243 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.393255 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.393257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.393416 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:34.393766 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:34.393771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:34.393775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:34.394077 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:34.394377 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:34.394680 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 02:38:34.394934 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":328,"threads_started":1}
W20260621 02:38:34.396613 3906 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52056
W20260621 02:38:34.397785 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3783: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:38:34.397974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 3783
I20260621 02:38:34.413094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3423
I20260621 02:38:34.414297 3485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:34.514673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 3423
2026-06-21T02:38:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3548 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3928](info): set up 2 sockets
Jun 21 02:38:34 dist-test-slave-jkn7 krb5kdc[3928](info): commencing operation
krb5kdc: starting...
W20260621 02:38:35.601756 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.009s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:35 dist-test-slave-jkn7 krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:35.794915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43585
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:36.100281 3941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:36.100605 3941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:36.100768 3941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:36.107728 3941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:36.107862 3941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:36.107967 3941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:36.108072 3941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:36.108140 3941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:36.117450 3941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:36.118659 3941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:36.119894 3941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:36.130985 3946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:36.130966 3947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:36.131199 3949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:36.131899 3941 server_base.cc:1034] running on GCE node
I20260621 02:38:36.132655 3941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:36.133999 3941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:36.135219 3941 hybrid_clock.cc:648] HybridClock initialized: now 1782009516135183 us; error 48 us; skew 500 ppm
I20260621 02:38:36.135547 3941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:36 dist-test-slave-jkn7 krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:36.142583 3941 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:36.144073 3941 webserver.cc:460] Webserver started at http://127.0.97.62:38351/ using document root <none> and password file <none>
I20260621 02:38:36.144791 3941 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:36.144932 3941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:36.145241 3941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:36.147996 3941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "aa28869921314b41ba7c5e3afece37c3"
format_stamp: "Formatted at 2026-06-21 02:38:36 on dist-test-slave-jkn7"
server_key: "a5355b03b97ff0b212154f43491fa7c6"
server_key_iv: "3778326dbed948b2f64df8e907d7fd4b"
server_key_version: "encryptionkey@0"
I20260621 02:38:36.148758 3941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "aa28869921314b41ba7c5e3afece37c3"
format_stamp: "Formatted at 2026-06-21 02:38:36 on dist-test-slave-jkn7"
server_key: "a5355b03b97ff0b212154f43491fa7c6"
server_key_iv: "3778326dbed948b2f64df8e907d7fd4b"
server_key_version: "encryptionkey@0"
I20260621 02:38:36.153571 3941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20260621 02:38:36.156720 3956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:36.158037 3941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:36.158236 3941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "aa28869921314b41ba7c5e3afece37c3"
format_stamp: "Formatted at 2026-06-21 02:38:36 on dist-test-slave-jkn7"
server_key: "a5355b03b97ff0b212154f43491fa7c6"
server_key_iv: "3778326dbed948b2f64df8e907d7fd4b"
server_key_version: "encryptionkey@0"
I20260621 02:38:36.158427 3941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:36.178198 3941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:36.181820 3941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:36.182087 3941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:36.199940 3941 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43585
I20260621 02:38:36.199970 4007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43585 every 8 connection(s)
I20260621 02:38:36.201241 3941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:36.204613 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:36.210451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 3941
I20260621 02:38:36.210680 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:36.211283 388 external_mini_cluster.cc:1428] Setting key 8f1f71299355da98383f656963358dec
I20260621 02:38:36.214577 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3: Bootstrap starting.
I20260621 02:38:36.217873 4008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:36.218920 4008 log.cc:824] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:36.221486 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3: No bootstrap required, opened a new log
Jun 21 02:38:36 dist-test-slave-jkn7 krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:36.229327 4008 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28869921314b41ba7c5e3afece37c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } }
I20260621 02:38:36.229683 4008 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:36.229830 4008 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa28869921314b41ba7c5e3afece37c3, State: Initialized, Role: FOLLOWER
I20260621 02:38:36.230394 4008 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28869921314b41ba7c5e3afece37c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } }
I20260621 02:38:36.230651 4008 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:36.230841 4008 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:36.231014 4008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:36.232573 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:36.213125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.0.97.62:43585)
0621 02:38:36.213891 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:36.213903 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:36.213943 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:38:36.215621 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:36.215643 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:36.216018 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:36.216323 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:36.218141 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:36.219446 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:36.220718 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:36.221408 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:36.221606 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:36.225486 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:36.225494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:36.225510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:36.225549 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:36.228539 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:36.229184 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:36.229191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:36.229200 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:36.229407 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:36.229848 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:36.229853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:36.229858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:36.230492 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:36.230793 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:36.231245 (+ 452us) server_negotiation.cc:299] Negotiation successful
0621 02:38:36.231633 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":388,"threads_started":1}
I20260621 02:38:36.233455 4008 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28869921314b41ba7c5e3afece37c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } }
I20260621 02:38:36.233956 4008 leader_election.cc:302] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa28869921314b41ba7c5e3afece37c3; no voters:
I20260621 02:38:36.234504 4008 leader_election.cc:288] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:36.234643 4013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:36.235656 4013 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [term 1 LEADER]: Becoming Leader. State: Replica: aa28869921314b41ba7c5e3afece37c3, State: Running, Role: LEADER
I20260621 02:38:36.236217 4013 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28869921314b41ba7c5e3afece37c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } }
I20260621 02:38:36.236872 4008 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:36.238788 4015 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa28869921314b41ba7c5e3afece37c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28869921314b41ba7c5e3afece37c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } }
I20260621 02:38:36.239166 4015 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:36.239266 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa28869921314b41ba7c5e3afece37c3. Latest consensus state: current_term: 1 leader_uuid: "aa28869921314b41ba7c5e3afece37c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28869921314b41ba7c5e3afece37c3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } }
I20260621 02:38:36.239586 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:36.241379 4018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:36.246332 4018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:36.255906 4018 catalog_manager.cc:1347] Generated new cluster ID: 58cc67f830774dfca0a332f7e6e80a0e
I20260621 02:38:36.256048 4018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:36.267019 4018 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:36.268172 4018 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:36.275374 4018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa28869921314b41ba7c5e3afece37c3: Generated new TSK 0
I20260621 02:38:36.276075 4018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:38:36.326236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43585
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:36.664610 4034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:36.664870 4034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:36.665027 4034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:36.672003 4034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:36.672145 4034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:36.672334 4034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:36.681622 4034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:36.682852 4034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:36.684152 4034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:36.697000 4039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:36.697026 4040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:36.698298 4034 server_base.cc:1034] running on GCE node
W20260621 02:38:36.698500 4042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:36.699272 4034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:36.700431 4034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:36.701624 4034 hybrid_clock.cc:648] HybridClock initialized: now 1782009516701585 us; error 30 us; skew 500 ppm
I20260621 02:38:36.701930 4034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:36 dist-test-slave-jkn7 krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:36.705049 4034 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:38:36.706382 4034 webserver.cc:460] Webserver started at http://127.0.97.1:46455/ using document root <none> and password file <none>
I20260621 02:38:36.706983 4034 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:36.707077 4034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:36.707326 4034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:36.710086 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "5f0165766447410baf75aa61f7b0e0e9"
format_stamp: "Formatted at 2026-06-21 02:38:36 on dist-test-slave-jkn7"
server_key: "f61f2701460b32fb363091b7bda00931"
server_key_iv: "0d5809eaad67ca7b716b1d86c13cdbcd"
server_key_version: "encryptionkey@0"
I20260621 02:38:36.710778 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5f0165766447410baf75aa61f7b0e0e9"
format_stamp: "Formatted at 2026-06-21 02:38:36 on dist-test-slave-jkn7"
server_key: "f61f2701460b32fb363091b7bda00931"
server_key_iv: "0d5809eaad67ca7b716b1d86c13cdbcd"
server_key_version: "encryptionkey@0"
I20260621 02:38:36.715588 4034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:38:36.718840 4049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:36.720078 4034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 02:38:36.720259 4034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "5f0165766447410baf75aa61f7b0e0e9"
format_stamp: "Formatted at 2026-06-21 02:38:36 on dist-test-slave-jkn7"
server_key: "f61f2701460b32fb363091b7bda00931"
server_key_iv: "0d5809eaad67ca7b716b1d86c13cdbcd"
server_key_version: "encryptionkey@0"
I20260621 02:38:36.720467 4034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:36.736444 4034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:36.740341 4034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:36.740604 4034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:36.741534 4034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:36.742738 4034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:36.742806 4034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:36.742964 4034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:36.743022 4034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:36.775892 4034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45641
I20260621 02:38:36.775991 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45641 every 8 connection(s)
I20260621 02:38:36.777158 4034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:36.785704 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 4034
I20260621 02:38:36.785914 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:36.786525 388 external_mini_cluster.cc:1428] Setting key dc350d2b6c2118d11c1abb9d978a231b
Jun 21 02:38:36 dist-test-slave-jkn7 krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:36.797122 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:36.779622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45803 (local address 127.0.97.62:43585)
0621 02:38:36.780054 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:36.780061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:36.781046 (+ 985us) server_negotiation.cc:407] Connection header received
0621 02:38:36.782778 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:36.782782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:36.782847 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:36.782969 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:36.784292 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:36.784998 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:36.786628 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:36.787200 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:36.787306 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:36.790810 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:36.790820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:36.790827 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:36.790870 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:36.794055 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:36.794968 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:36.794972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:36.794974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:36.795076 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:36.795544 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:36.795547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:36.795549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:36.795783 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:36.795885 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:36.796732 (+ 847us) server_negotiation.cc:299] Negotiation successful
0621 02:38:36.796875 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":210,"threads_started":1}
I20260621 02:38:36.798578 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:36.780213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43585 (local address 127.0.97.1:45803)
0621 02:38:36.780888 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:36.780930 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:36.782534 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:36.783159 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:36.783166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:36.783637 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:36.784144 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:36.784159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:36.785141 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:36.785144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:36.786488 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:36.786498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:36.787465 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:36.787472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:36.787703 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:36.788522 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:36.788547 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:36.790589 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:36.794202 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:36.794213 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:36.794230 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:36.794850 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:36.795175 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:36.795183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:36.795189 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:36.795432 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:36.795901 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:36.795912 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:36.796349 (+ 437us) client_negotiation.cc:769] Sending connection context
0621 02:38:36.796644 (+ 295us) client_negotiation.cc:240] Negotiation successful
0621 02:38:36.797168 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":279,"threads_started":1}
I20260621 02:38:36.800069 4162 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43585
I20260621 02:38:36.800360 4162 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:36.801038 4162 heartbeater.cc:511] Master 127.0.97.62:43585 requested a full tablet report, sending...
I20260621 02:38:36.803129 3973 ts_manager.cc:194] Registered new tserver with Master: 5f0165766447410baf75aa61f7b0e0e9 (127.0.97.1:45641)
I20260621 02:38:36.804520 3973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45803
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:38:36.827745 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43585
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:37.139838 4168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:37.140177 4168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:37.140338 4168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:37.147634 4168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:37.147786 4168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:37.147957 4168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:37.157624 4168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:37.158907 4168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:37.160205 4168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:37.173355 4173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:37.173422 4176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:37.173418 4174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:37.174520 4168 server_base.cc:1034] running on GCE node
I20260621 02:38:37.175117 4168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:37.176029 4168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:37.177263 4168 hybrid_clock.cc:648] HybridClock initialized: now 1782009517177206 us; error 76 us; skew 500 ppm
I20260621 02:38:37.177634 4168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:37 dist-test-slave-jkn7 krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:37.180608 4168 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:38:37.182073 4168 webserver.cc:460] Webserver started at http://127.0.97.2:46349/ using document root <none> and password file <none>
I20260621 02:38:37.182742 4168 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:37.182886 4168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:37.183187 4168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:37.186139 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "2d15958c30e140ef945d61dd3dda14f0"
format_stamp: "Formatted at 2026-06-21 02:38:37 on dist-test-slave-jkn7"
server_key: "55751e282aec9809b2fae7bf4eae2a48"
server_key_iv: "0624fdd7e1ebc2c68fc4adacd16c0713"
server_key_version: "encryptionkey@0"
I20260621 02:38:37.186888 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2d15958c30e140ef945d61dd3dda14f0"
format_stamp: "Formatted at 2026-06-21 02:38:37 on dist-test-slave-jkn7"
server_key: "55751e282aec9809b2fae7bf4eae2a48"
server_key_iv: "0624fdd7e1ebc2c68fc4adacd16c0713"
server_key_version: "encryptionkey@0"
I20260621 02:38:37.191800 4168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:38:37.195109 4183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:37.196305 4168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:38:37.196473 4168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "2d15958c30e140ef945d61dd3dda14f0"
format_stamp: "Formatted at 2026-06-21 02:38:37 on dist-test-slave-jkn7"
server_key: "55751e282aec9809b2fae7bf4eae2a48"
server_key_iv: "0624fdd7e1ebc2c68fc4adacd16c0713"
server_key_version: "encryptionkey@0"
I20260621 02:38:37.196676 4168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:37.260835 4168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:37.265023 4168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:37.265347 4168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:37.266285 4168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:37.267609 4168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:37.267714 4168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:37.267872 4168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:37.267946 4168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:37.300783 4168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36141
I20260621 02:38:37.300807 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36141 every 8 connection(s)
I20260621 02:38:37.302148 4168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:37.304399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 4168
I20260621 02:38:37.304605 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:37.305140 388 external_mini_cluster.cc:1428] Setting key 7f5f340200c6b22398d0cd9564840062
Jun 21 02:38:37 dist-test-slave-jkn7 krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:38:37.323560 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:37.304751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58065 (local address 127.0.97.62:43585)
0621 02:38:37.305149 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:37.305154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:37.306524 (+ 1370us) server_negotiation.cc:407] Connection header received
0621 02:38:37.308302 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:37.308311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:37.308400 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:37.308525 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:37.309980 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.311128 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.313086 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.313538 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.313633 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:37.317813 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:37.317819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:37.317822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:37.317853 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:37.320475 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.321216 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.321222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.321232 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.321414 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.321834 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.321839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.321842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.322094 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:37.322249 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:37.322918 (+ 669us) server_negotiation.cc:299] Negotiation successful
0621 02:38:37.323126 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":210,"threads_started":1}
I20260621 02:38:37.324658 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:37.305455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43585 (local address 127.0.97.2:58065)
0621 02:38:37.306357 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:37.306402 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:37.308037 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:37.308716 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:37.308724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:37.309194 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:37.309814 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:37.309831 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.311288 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.311291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:37.312917 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:37.312936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.313754 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.313764 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:37.314077 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:37.314791 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:37.314829 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:37.317592 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:37.320619 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:37.320628 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:37.320645 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:37.321070 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:37.321523 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:37.321529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:37.321533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:37.321715 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:37.322223 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:37.322231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:37.322561 (+ 330us) client_negotiation.cc:769] Sending connection context
0621 02:38:37.322832 (+ 271us) client_negotiation.cc:240] Negotiation successful
0621 02:38:37.323437 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":415,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260621 02:38:37.326193 4296 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43585
I20260621 02:38:37.326514 4296 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:37.327409 4296 heartbeater.cc:511] Master 127.0.97.62:43585 requested a full tablet report, sending...
I20260621 02:38:37.328970 3973 ts_manager.cc:194] Registered new tserver with Master: 2d15958c30e140ef945d61dd3dda14f0 (127.0.97.2:36141)
I20260621 02:38:37.329828 3973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:38:37.348027 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43585
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:37.667408 4302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:37.667721 4302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:37.667886 4302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:37.675004 4302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:37.675135 4302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:37.675307 4302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:37.684484 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:37.685710 4302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:37.686987 4302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:37.699486 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:37.699535 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:37.699559 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:37.705699 4302 server_base.cc:1034] running on GCE node
I20260621 02:38:37.706270 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:37.707142 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:37.708328 4302 hybrid_clock.cc:648] HybridClock initialized: now 1782009517708273 us; error 49 us; skew 500 ppm
I20260621 02:38:37.708588 4302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:37 dist-test-slave-jkn7 krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:37.711508 4302 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:38:37.712941 4302 webserver.cc:460] Webserver started at http://127.0.97.3:35033/ using document root <none> and password file <none>
I20260621 02:38:37.713685 4302 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:37.713835 4302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:37.714130 4302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:37.716939 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "bd88484d51274f6794db4be779749358"
format_stamp: "Formatted at 2026-06-21 02:38:37 on dist-test-slave-jkn7"
server_key: "7565833a8a26a47b1b2c9b3af7840f44"
server_key_iv: "5fcf4fe3da462865ebc689aabf9cbf72"
server_key_version: "encryptionkey@0"
I20260621 02:38:37.717767 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bd88484d51274f6794db4be779749358"
format_stamp: "Formatted at 2026-06-21 02:38:37 on dist-test-slave-jkn7"
server_key: "7565833a8a26a47b1b2c9b3af7840f44"
server_key_iv: "5fcf4fe3da462865ebc689aabf9cbf72"
server_key_version: "encryptionkey@0"
I20260621 02:38:37.722914 4302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:38:37.726689 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:37.728018 4302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:38:37.728204 4302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "bd88484d51274f6794db4be779749358"
format_stamp: "Formatted at 2026-06-21 02:38:37 on dist-test-slave-jkn7"
server_key: "7565833a8a26a47b1b2c9b3af7840f44"
server_key_iv: "5fcf4fe3da462865ebc689aabf9cbf72"
server_key_version: "encryptionkey@0"
I20260621 02:38:37.728430 4302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:37.760740 4302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:37.764891 4302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:37.765180 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:37.766177 4302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:37.767432 4302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:37.767535 4302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:37.767678 4302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:37.767760 4302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:37.801486 4302 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42651
I20260621 02:38:37.801519 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42651 every 8 connection(s)
I20260621 02:38:37.802762 4302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:37.807021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 4302
I20260621 02:38:37.807212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:37.807518 4162 heartbeater.cc:503] Master 127.0.97.62:43585 was elected leader, sending a full tablet report...
I20260621 02:38:37.807785 388 external_mini_cluster.cc:1428] Setting key 5f4fa910a00c8e513106b110ddae256e
Jun 21 02:38:37 dist-test-slave-jkn7 krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:37.822265 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:37.805195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33051 (local address 127.0.97.62:43585)
0621 02:38:37.805395 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:37.805400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:37.806521 (+ 1121us) server_negotiation.cc:407] Connection header received
0621 02:38:37.809017 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:37.809023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:37.809113 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:37.809218 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:37.810863 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.811588 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.813086 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.813542 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.813637 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:37.816920 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:37.816926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:37.816929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:37.816961 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:37.819337 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.820054 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.820058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.820060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.820170 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.820530 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.820533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.820535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.820780 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:37.820883 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:37.821724 (+ 841us) server_negotiation.cc:299] Negotiation successful
0621 02:38:37.821972 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:38:37.823534 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:37.805665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43585 (local address 127.0.97.3:33051)
0621 02:38:37.806357 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:37.806397 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:37.808741 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:37.809504 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:37.809517 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:37.810137 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:37.810698 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:37.810714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.811722 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.811726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:37.812968 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:37.812979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.813720 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.813724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:37.813914 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:37.814600 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:37.814623 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:37.816718 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:37.819487 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:37.819499 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:37.819513 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:37.819953 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:37.820267 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:37.820271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:37.820273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:37.820445 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:37.820887 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:37.820893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:37.821251 (+ 358us) client_negotiation.cc:769] Sending connection context
0621 02:38:37.821637 (+ 386us) client_negotiation.cc:240] Negotiation successful
0621 02:38:37.822131 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":285,"threads_started":1}
I20260621 02:38:37.824831 4430 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43585
I20260621 02:38:37.825160 4430 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:37.825922 4430 heartbeater.cc:511] Master 127.0.97.62:43585 requested a full tablet report, sending...
I20260621 02:38:37.827230 3973 ts_manager.cc:194] Registered new tserver with Master: bd88484d51274f6794db4be779749358 (127.0.97.3:42651)
I20260621 02:38:37.827965 3973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33051
I20260621 02:38:37.835803 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:38:37 dist-test-slave-jkn7 krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:38:37 dist-test-slave-jkn7 krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:37.884311 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:37.869991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.0.97.1:45641)
0621 02:38:37.870580 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:37.870588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:37.870616 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:38:37.870676 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:37.870682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:37.870908 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:37.871037 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:37.872323 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.873372 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.874541 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.875079 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.875170 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:37.878847 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:37.878857 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:37.878871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:37.878906 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:37.881789 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.882437 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.882444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.882447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.882558 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.882876 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.882879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.882881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.883190 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:37.883404 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:37.883728 (+ 324us) server_negotiation.cc:299] Negotiation successful
0621 02:38:37.883913 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":309,"threads_started":1}
Jun 21 02:38:37 dist-test-slave-jkn7 krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:37.901432 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:37.888433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.0.97.62:43585)
0621 02:38:37.888657 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:37.888662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:37.888756 (+ 94us) server_negotiation.cc:407] Connection header received
0621 02:38:37.888873 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:37.888877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:37.888939 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:37.889016 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:37.890223 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.890958 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.892174 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.892582 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.892679 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:37.896448 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:37.896453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:37.896456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:37.896488 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:37.898851 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.899474 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.899478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.899480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.899644 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.899945 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.899949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.899950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.900306 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:37.900484 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:37.900836 (+ 352us) server_negotiation.cc:299] Negotiation successful
0621 02:38:37.900994 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260621 02:38:37.901988 3973 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50274
I20260621 02:38:37.902921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 4034
I20260621 02:38:37.914471 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:37.903688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.0.97.1:45641)
0621 02:38:37.903835 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:37.903840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:37.904117 (+ 277us) server_negotiation.cc:407] Connection header received
0621 02:38:37.904238 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:37.904242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:37.904300 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:37.904373 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:37.905540 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.906206 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.907372 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:37.907785 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:37.907933 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:37.910091 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:37.910095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:37.910099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:37.910129 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:37.912444 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.912994 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.912998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.913000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.913112 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:37.913472 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:37.913475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:37.913477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:37.913698 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:37.913882 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:37.914103 (+ 221us) server_negotiation.cc:299] Negotiation successful
0621 02:38:37.914252 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:38:37.915098 4157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:37.997901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 4034
I20260621 02:38:38.013744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 4168
Jun 21 02:38:38 dist-test-slave-jkn7 krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:38:38.029181 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:38.014628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.0.97.2:36141)
0621 02:38:38.015201 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:38.015208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:38.015235 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:38:38.015303 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:38.015309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:38.015517 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:38.015651 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:38.016742 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:38.017878 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:38.019101 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:38.019636 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:38.019749 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:38.023756 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:38.023767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:38.023781 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:38.023811 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:38.026518 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:38.027169 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:38.027176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:38.027178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:38.027317 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:38.027696 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:38.027700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:38.027701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:38.028034 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:38.028260 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:38.028547 (+ 287us) server_negotiation.cc:299] Negotiation successful
0621 02:38:38.028737 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":325,"threads_started":1}
I20260621 02:38:38.030910 4291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:38.112591 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 4168
I20260621 02:38:38.128226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 4302
Jun 21 02:38:38 dist-test-slave-jkn7 krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:38:38.143440 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:38.129050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.0.97.3:42651)
0621 02:38:38.129681 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:38.129688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:38.129722 (+ 34us) server_negotiation.cc:407] Connection header received
0621 02:38:38.129781 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:38.129787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:38.129993 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:38.130115 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:38.131289 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:38.132464 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:38.133826 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:38.134377 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:38.134485 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:38.138537 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:38.138547 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:38.138561 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:38.138588 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:38.141075 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:38.141751 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:38.141758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:38.141760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:38.141875 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:38.142170 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:38.142174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:38.142175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:38.142469 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:38.142738 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:38.142899 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 02:38:38.143110 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":325,"threads_started":1}
I20260621 02:38:38.144596 4425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:38.229099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 4302
I20260621 02:38:38.244596 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 3941
I20260621 02:38:38.245785 4003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:38.344527 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 3941
2026-06-21T02:38:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3828 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:38 dist-test-slave-jkn7 krb5kdc[4449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:38 dist-test-slave-jkn7 krb5kdc[4449](info): set up 2 sockets
Jun 21 02:38:38 dist-test-slave-jkn7 krb5kdc[4449](info): commencing operation
krb5kdc: starting...
W20260621 02:38:39.441746 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:39 dist-test-slave-jkn7 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:39.622355 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36147
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:39.931629 4462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:39.931900 4462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:39.932080 4462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:39.938966 4462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:39.939064 4462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:39.939169 4462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:39.939265 4462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:39.939358 4462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:39.949112 4462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:39.950342 4462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:39.951601 4462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:39.962749 4468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:39.962702 4467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:39.962774 4470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:39.963726 4462 server_base.cc:1034] running on GCE node
I20260621 02:38:39.964395 4462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:39.965806 4462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:39.967022 4462 hybrid_clock.cc:648] HybridClock initialized: now 1782009519966960 us; error 50 us; skew 500 ppm
I20260621 02:38:39.967329 4462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:39 dist-test-slave-jkn7 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:39.970292 4462 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:39.971433 4462 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:39.971544 4462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:39.971880 4462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:39.974748 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "909e228a2e23488e8f82089f8d2fbfc8"
format_stamp: "Formatted at 2026-06-21 02:38:39 on dist-test-slave-jkn7"
server_key: "27de904b740ebe70ee6f6727654210d6"
server_key_iv: "16e7a932a3986a6406d9a8ced29e5fac"
server_key_version: "encryptionkey@0"
I20260621 02:38:39.975529 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "909e228a2e23488e8f82089f8d2fbfc8"
format_stamp: "Formatted at 2026-06-21 02:38:39 on dist-test-slave-jkn7"
server_key: "27de904b740ebe70ee6f6727654210d6"
server_key_iv: "16e7a932a3986a6406d9a8ced29e5fac"
server_key_version: "encryptionkey@0"
I20260621 02:38:39.980283 4462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260621 02:38:39.983753 4476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:39.985215 4462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:39.985463 4462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "909e228a2e23488e8f82089f8d2fbfc8"
format_stamp: "Formatted at 2026-06-21 02:38:39 on dist-test-slave-jkn7"
server_key: "27de904b740ebe70ee6f6727654210d6"
server_key_iv: "16e7a932a3986a6406d9a8ced29e5fac"
server_key_version: "encryptionkey@0"
I20260621 02:38:39.985697 4462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:40.009553 4462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:40.013818 4462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:40.014107 4462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:40.032114 4462 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36147
I20260621 02:38:40.032137 4527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36147 every 8 connection(s)
I20260621 02:38:40.033022 4462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:40.036576 4528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:40.039451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 4462
I20260621 02:38:40.039691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:40.040302 388 external_mini_cluster.cc:1428] Setting key 0df4ba615e24945ac4454d0d4f683afc
I20260621 02:38:40.048650 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8: Bootstrap starting.
I20260621 02:38:40.052361 4528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:40.053681 4528 log.cc:824] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:38:40 dist-test-slave-jkn7 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:40.056384 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8: No bootstrap required, opened a new log
I20260621 02:38:40.062705 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:40.042228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.0.97.62:36147)
0621 02:38:40.042983 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:40.042994 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:40.043033 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:38:40.044569 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:40.044590 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:40.044972 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:40.045344 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:40.047094 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:40.048491 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:40.050236 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:40.050888 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:40.051010 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:40.055809 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:40.055817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:40.055831 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:40.055867 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:40.058912 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:40.059576 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:40.059582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:40.059587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:40.059738 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:40.060125 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:40.060129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:40.060134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:40.060633 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:40.060957 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:40.061370 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 02:38:40.061759 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":346,"threads_started":1}
I20260621 02:38:40.063982 4528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e228a2e23488e8f82089f8d2fbfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36147 } }
I20260621 02:38:40.064343 4528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:40.064460 4528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909e228a2e23488e8f82089f8d2fbfc8, State: Initialized, Role: FOLLOWER
I20260621 02:38:40.065060 4528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e228a2e23488e8f82089f8d2fbfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36147 } }
I20260621 02:38:40.065306 4528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:40.065445 4528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:40.065629 4528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:40.067626 4528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e228a2e23488e8f82089f8d2fbfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36147 } }
I20260621 02:38:40.068090 4528 leader_election.cc:302] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 909e228a2e23488e8f82089f8d2fbfc8; no voters:
I20260621 02:38:40.068707 4528 leader_election.cc:288] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:40.068810 4533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:40.069514 4533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [term 1 LEADER]: Becoming Leader. State: Replica: 909e228a2e23488e8f82089f8d2fbfc8, State: Running, Role: LEADER
I20260621 02:38:40.070063 4533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e228a2e23488e8f82089f8d2fbfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36147 } }
I20260621 02:38:40.070828 4528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:40.072800 4534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "909e228a2e23488e8f82089f8d2fbfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e228a2e23488e8f82089f8d2fbfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36147 } } }
I20260621 02:38:40.073210 4534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:40.072903 4535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 909e228a2e23488e8f82089f8d2fbfc8. Latest consensus state: current_term: 1 leader_uuid: "909e228a2e23488e8f82089f8d2fbfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e228a2e23488e8f82089f8d2fbfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36147 } } }
I20260621 02:38:40.073380 4535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:40.074702 4538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:40.079941 4538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:40.088968 4538 catalog_manager.cc:1347] Generated new cluster ID: 951ccc5a10d747bba6c2bfe13642c79d
I20260621 02:38:40.089100 4538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:40.102761 4538 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:40.103960 4538 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:40.117787 4538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8: Generated new TSK 0
I20260621 02:38:40.118731 4538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:38:40.169709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36147
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:40.500288 4554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:40.500592 4554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:40.500757 4554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:40.508155 4554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:40.508275 4554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:40.508460 4554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:40.518541 4554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:40.519862 4554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:40.521175 4554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:40.533747 4559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:40.533824 4560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:40.533849 4562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:40.534854 4554 server_base.cc:1034] running on GCE node
I20260621 02:38:40.535451 4554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:40.536435 4554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:40.537664 4554 hybrid_clock.cc:648] HybridClock initialized: now 1782009520537619 us; error 26 us; skew 500 ppm
I20260621 02:38:40.538026 4554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:40 dist-test-slave-jkn7 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:40.541210 4554 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:38:40.542441 4554 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:40.542587 4554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:40.542925 4554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:40.546015 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b"
format_stamp: "Formatted at 2026-06-21 02:38:40 on dist-test-slave-jkn7"
server_key: "daf67921be1b810989c727f10ff40468"
server_key_iv: "ad933d9c59c451d379fdf2dc6d7d1eb9"
server_key_version: "encryptionkey@0"
I20260621 02:38:40.546850 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b"
format_stamp: "Formatted at 2026-06-21 02:38:40 on dist-test-slave-jkn7"
server_key: "daf67921be1b810989c727f10ff40468"
server_key_iv: "ad933d9c59c451d379fdf2dc6d7d1eb9"
server_key_version: "encryptionkey@0"
I20260621 02:38:40.551777 4554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260621 02:38:40.555351 4568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:40.556646 4554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:40.556826 4554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b"
format_stamp: "Formatted at 2026-06-21 02:38:40 on dist-test-slave-jkn7"
server_key: "daf67921be1b810989c727f10ff40468"
server_key_iv: "ad933d9c59c451d379fdf2dc6d7d1eb9"
server_key_version: "encryptionkey@0"
I20260621 02:38:40.557036 4554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:40.579325 4554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:40.583153 4554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:40.583420 4554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:40.584336 4554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:40.585668 4554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:40.585757 4554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:40.585827 4554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:40.585881 4554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:40.617848 4554 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40061
I20260621 02:38:40.617869 4680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40061 every 8 connection(s)
I20260621 02:38:40.618757 4554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:40.627432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 4554
I20260621 02:38:40.627640 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:40.628237 388 external_mini_cluster.cc:1428] Setting key f0dc530b9431ab23a3ed0ddb25de2e42
Jun 21 02:38:40 dist-test-slave-jkn7 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:40.638774 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:40.621264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56173 (local address 127.0.97.62:36147)
0621 02:38:40.621759 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:40.621763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:40.622681 (+ 918us) server_negotiation.cc:407] Connection header received
0621 02:38:40.624503 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:40.624509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:40.624603 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:40.624723 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:40.626307 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:40.627042 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:40.628847 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:40.629462 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:40.629607 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:40.633029 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:40.633035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:40.633038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:40.633068 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:40.635620 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:40.636518 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:40.636522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:40.636524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:40.636629 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:40.637078 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:40.637081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:40.637083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:40.637394 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:40.637509 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:40.638368 (+ 859us) server_negotiation.cc:299] Negotiation successful
0621 02:38:40.638528 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":245,"threads_started":1}
I20260621 02:38:40.640381 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:40.621809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36147 (local address 127.0.97.1:56173)
0621 02:38:40.622514 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:40.622554 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:40.624221 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:40.624966 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:40.624974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:40.625560 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:40.626131 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:40.626148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:40.627218 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:40.627223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:40.628670 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:40.628682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:40.629692 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:40.629701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:40.629986 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:40.630740 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:40.630761 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:40.632813 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:40.635756 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:40.635768 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:40.635787 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:40.636404 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:40.636730 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:40.636738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:40.636744 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:40.636978 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:40.637533 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:40.637544 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:40.637990 (+ 446us) client_negotiation.cc:769] Sending connection context
0621 02:38:40.638289 (+ 299us) client_negotiation.cc:240] Negotiation successful
0621 02:38:40.638859 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":284,"threads_started":1}
I20260621 02:38:40.642068 4681 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36147
I20260621 02:38:40.642383 4681 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:40.643082 4681 heartbeater.cc:511] Master 127.0.97.62:36147 requested a full tablet report, sending...
I20260621 02:38:40.645118 4493 ts_manager.cc:194] Registered new tserver with Master: 89c5a3a732a3488ab3d6b1a8174cbf3b (127.0.97.1:40061)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260621 02:38:40.646605 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:38:40.672519 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36147
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:40.984174 4687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:40.984473 4687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:40.984632 4687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:40.991572 4687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:40.991714 4687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:40.991878 4687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:41.000988 4687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:41.002257 4687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:41.003474 4687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:41.015535 4693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:41.015707 4695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:41.015719 4692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:41.016640 4687 server_base.cc:1034] running on GCE node
I20260621 02:38:41.017172 4687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:41.018101 4687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:41.019330 4687 hybrid_clock.cc:648] HybridClock initialized: now 1782009521019287 us; error 44 us; skew 500 ppm
I20260621 02:38:41.019635 4687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:41 dist-test-slave-jkn7 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:41.022632 4687 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:38:41.023698 4687 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:41.023828 4687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:41.024127 4687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:41.026851 4687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "73117de690ad4d528fc8fbc508e94716"
format_stamp: "Formatted at 2026-06-21 02:38:41 on dist-test-slave-jkn7"
server_key: "17bcd1383d146f2335ae6fb97ded661b"
server_key_iv: "7b474341387085fc1dee6f1c03f880c5"
server_key_version: "encryptionkey@0"
I20260621 02:38:41.027623 4687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "73117de690ad4d528fc8fbc508e94716"
format_stamp: "Formatted at 2026-06-21 02:38:41 on dist-test-slave-jkn7"
server_key: "17bcd1383d146f2335ae6fb97ded661b"
server_key_iv: "7b474341387085fc1dee6f1c03f880c5"
server_key_version: "encryptionkey@0"
I20260621 02:38:41.032505 4687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260621 02:38:41.036484 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:41.037986 4687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 02:38:41.038134 4687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "73117de690ad4d528fc8fbc508e94716"
format_stamp: "Formatted at 2026-06-21 02:38:41 on dist-test-slave-jkn7"
server_key: "17bcd1383d146f2335ae6fb97ded661b"
server_key_iv: "7b474341387085fc1dee6f1c03f880c5"
server_key_version: "encryptionkey@0"
I20260621 02:38:41.038353 4687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:41.076424 4687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:41.080926 4687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:41.081151 4687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:41.082152 4687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:41.083424 4687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:41.083509 4687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:41.083590 4687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:41.083643 4687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:41.117071 4687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36899
I20260621 02:38:41.117126 4813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36899 every 8 connection(s)
I20260621 02:38:41.117971 4687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:41.120404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 4687
I20260621 02:38:41.120596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:41.121121 388 external_mini_cluster.cc:1428] Setting key 3d96fb12173e45091f84459357c74c31
Jun 21 02:38:41 dist-test-slave-jkn7 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:38:41.139135 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.120525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44371 (local address 127.0.97.62:36147)
0621 02:38:41.120737 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.120745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.122306 (+ 1561us) server_negotiation.cc:407] Connection header received
0621 02:38:41.124179 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.124185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.124277 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.124385 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:41.125902 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.126750 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.128886 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.129349 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.129479 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.132694 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:41.132699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:41.132702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:41.132731 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:41.136033 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:41.136711 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:41.136717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:41.136723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:41.136884 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:41.137265 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:41.137271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:41.137275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:41.137654 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:41.137825 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:41.138490 (+ 665us) server_negotiation.cc:299] Negotiation successful
0621 02:38:41.138708 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:38:41.139846 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.121187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36147 (local address 127.0.97.2:44371)
0621 02:38:41.122137 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:41.122183 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:41.123920 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:41.124605 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:41.124614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:41.125103 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:41.125726 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.125751 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.126915 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.126923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.128716 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.128737 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.129559 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.129566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.129778 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.130452 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:41.130473 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:41.132485 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:41.136169 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:41.136175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:41.136187 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:41.136589 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:41.136994 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:41.136998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:41.137000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:41.137155 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:41.137791 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:41.137797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:41.138102 (+ 305us) client_negotiation.cc:769] Sending connection context
0621 02:38:41.138342 (+ 240us) client_negotiation.cc:240] Negotiation successful
0621 02:38:41.138698 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":314,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260621 02:38:41.141268 4814 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36147
I20260621 02:38:41.141580 4814 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:41.142390 4814 heartbeater.cc:511] Master 127.0.97.62:36147 requested a full tablet report, sending...
I20260621 02:38:41.144059 4493 ts_manager.cc:194] Registered new tserver with Master: 73117de690ad4d528fc8fbc508e94716 (127.0.97.2:36899)
I20260621 02:38:41.144842 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:38:41.164595 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36147
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:41.474202 4819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:41.474462 4819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:41.474555 4819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:41.482070 4819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:41.482162 4819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:41.482275 4819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:41.491318 4819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:41.492389 4819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:41.493566 4819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:41.505551 4825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:41.505635 4824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:41.505659 4827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:41.506237 4819 server_base.cc:1034] running on GCE node
I20260621 02:38:41.506803 4819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:41.507598 4819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:41.508843 4819 hybrid_clock.cc:648] HybridClock initialized: now 1782009521508787 us; error 44 us; skew 500 ppm
I20260621 02:38:41.509173 4819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:41 dist-test-slave-jkn7 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:41.516376 4819 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:38:41.517488 4819 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:41.517611 4819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:41.517910 4819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:41.520702 4819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "8ec10902c097453d96c11d6a7411362d"
format_stamp: "Formatted at 2026-06-21 02:38:41 on dist-test-slave-jkn7"
server_key: "fde55f7e88c06a119a476ca4ed41070c"
server_key_iv: "b044b967f1bdd3134c3b4254cd69c7ca"
server_key_version: "encryptionkey@0"
I20260621 02:38:41.521567 4819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8ec10902c097453d96c11d6a7411362d"
format_stamp: "Formatted at 2026-06-21 02:38:41 on dist-test-slave-jkn7"
server_key: "fde55f7e88c06a119a476ca4ed41070c"
server_key_iv: "b044b967f1bdd3134c3b4254cd69c7ca"
server_key_version: "encryptionkey@0"
I20260621 02:38:41.526126 4819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:38:41.529379 4833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:41.530900 4819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:38:41.531026 4819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "8ec10902c097453d96c11d6a7411362d"
format_stamp: "Formatted at 2026-06-21 02:38:41 on dist-test-slave-jkn7"
server_key: "fde55f7e88c06a119a476ca4ed41070c"
server_key_iv: "b044b967f1bdd3134c3b4254cd69c7ca"
server_key_version: "encryptionkey@0"
I20260621 02:38:41.531190 4819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:41.546105 4819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:41.550312 4819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:41.550516 4819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:41.551326 4819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:41.552459 4819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:41.552543 4819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:41.552608 4819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:41.552665 4819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:41.585126 4819 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33615
I20260621 02:38:41.585194 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33615 every 8 connection(s)
I20260621 02:38:41.585958 4819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:41.591066 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 4819
I20260621 02:38:41.591260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:41.591786 388 external_mini_cluster.cc:1428] Setting key d7cf7554a2ea403bb06d468ec76b2d26
Jun 21 02:38:41 dist-test-slave-jkn7 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:41.605542 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.588335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57545 (local address 127.0.97.62:36147)
0621 02:38:41.588529 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.588533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.589846 (+ 1313us) server_negotiation.cc:407] Connection header received
0621 02:38:41.592302 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.592308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.592393 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.592514 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:41.594341 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.595194 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.596720 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.597100 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.597199 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.600299 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:41.600304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:41.600307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:41.600340 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:41.602909 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:41.603538 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:41.603542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:41.603544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:41.603659 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:41.603988 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:41.603991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:41.603993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:41.604226 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:41.604339 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:41.604949 (+ 610us) server_negotiation.cc:299] Negotiation successful
0621 02:38:41.605156 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:38:41.606468 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.588863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36147 (local address 127.0.97.3:57545)
0621 02:38:41.589669 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:41.589711 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:41.591971 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:41.592761 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:41.592770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:41.593573 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:41.594164 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.594183 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.595349 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.595353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.596539 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.596551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.597338 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.597346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.597536 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.598119 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:41.598138 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:41.600113 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:41.603025 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:41.603038 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:41.603049 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:41.603437 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:41.603746 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:41.603749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:41.603751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:41.603898 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:41.604329 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:41.604334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:41.604605 (+ 271us) client_negotiation.cc:769] Sending connection context
0621 02:38:41.604852 (+ 247us) client_negotiation.cc:240] Negotiation successful
0621 02:38:41.605375 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":287,"threads_started":1}
I20260621 02:38:41.607944 4946 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36147
I20260621 02:38:41.608223 4946 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:41.608830 4946 heartbeater.cc:511] Master 127.0.97.62:36147 requested a full tablet report, sending...
I20260621 02:38:41.610214 4493 ts_manager.cc:194] Registered new tserver with Master: 8ec10902c097453d96c11d6a7411362d (127.0.97.3:33615)
I20260621 02:38:41.610956 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57545
I20260621 02:38:41.619138 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:38:41.636767 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.624416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.0.97.62:36147)
0621 02:38:41.624584 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.624606 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.625076 (+ 470us) server_negotiation.cc:407] Connection header received
0621 02:38:41.625262 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.625267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.625373 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.625475 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:41.626804 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.627485 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.628709 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.629138 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.629273 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.631486 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:41.631492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:41.631496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:41.631536 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:41.634107 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:41.634876 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:41.634884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:41.634889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:41.635007 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:41.635377 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:41.635383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:41.635386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:41.635686 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:41.635938 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:41.636264 (+ 326us) server_negotiation.cc:299] Negotiation successful
0621 02:38:41.636487 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:38:41.640501 4493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:38:41.647385 4493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:41.649833 4681 heartbeater.cc:503] Master 127.0.97.62:36147 was elected leader, sending a full tablet report...
I20260621 02:38:41.692720 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.661385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33615 (local address 127.0.0.1:36330)
0621 02:38:41.684507 (+ 23122us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:41.684523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:41.684623 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:41.685551 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:41.685559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:41.685589 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:41.685959 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.685971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.687865 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.687870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.689889 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.689900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.691667 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.691675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.691883 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.691944 (+ 61us) client_negotiation.cc:769] Sending connection context
0621 02:38:41.692073 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 02:38:41.692267 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23014,"thread_start_us":22541,"threads_started":1}
I20260621 02:38:41.693228 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.661660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.0.97.3:33615)
0621 02:38:41.681855 (+ 20195us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.681869 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.684729 (+ 2860us) server_negotiation.cc:407] Connection header received
0621 02:38:41.684833 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.684840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.685054 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.685249 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:41.686160 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.687697 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.690037 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.691362 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.692414 (+ 1052us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.692641 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:41.692733 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 02:38:41.692934 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19931,"thread_start_us":20223,"threads_started":1}
I20260621 02:38:41.693423 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.659131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40061 (local address 127.0.0.1:42880)
0621 02:38:41.683636 (+ 24505us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:41.683661 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:41.683830 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:41.686441 (+ 2611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:41.686446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:41.686470 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:41.686782 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.686791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.688590 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.688597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.690756 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.690771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.692581 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.692594 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.692780 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.692823 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 02:38:41.692900 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 02:38:41.693010 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24322,"mutex_wait_us":1,"thread_start_us":23173,"threads_started":1}
I20260621 02:38:41.693532 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.659226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.0.97.2:36899)
0621 02:38:41.683864 (+ 24638us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.683876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.683914 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:38:41.684039 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.684049 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.684381 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.684610 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:41.685740 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.687311 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.690151 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.691499 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.691623 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.691838 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:41.693002 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0621 02:38:41.693165 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24392,"thread_start_us":25472,"threads_started":1}
I20260621 02:38:41.692860 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.658895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36899 (local address 127.0.0.1:39202)
0621 02:38:41.682834 (+ 23939us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:41.682890 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:41.683144 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:41.684966 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:41.684974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:41.685001 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:41.685403 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.685416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.687452 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.687457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.689978 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.689992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.691981 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.691987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.692173 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.692205 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:38:41.692279 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 02:38:41.692426 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23577,"mutex_wait_us":4,"thread_start_us":25378,"threads_started":1}
I20260621 02:38:41.694804 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.659513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.0.97.1:40061)
0621 02:38:41.685984 (+ 26471us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.685993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.686020 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:38:41.686116 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.686124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.686268 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.686460 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:41.686941 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.688425 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.690935 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.692339 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.694034 (+ 1695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.694259 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:41.694340 (+ 81us) server_negotiation.cc:299] Negotiation successful
0621 02:38:41.694492 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26283,"thread_start_us":26110,"threads_started":1}
I20260621 02:38:41.697463 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 6d994812a1d841e8be1eb946654df000 (DEFAULT_TABLE table=test-table [id=33cdf7b2e80b478cabe4ce18b5b26017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:41.697463 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 6d994812a1d841e8be1eb946654df000 (DEFAULT_TABLE table=test-table [id=33cdf7b2e80b478cabe4ce18b5b26017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:41.697950 4616 tablet_service.cc:1467] Processing CreateTablet for tablet 6d994812a1d841e8be1eb946654df000 (DEFAULT_TABLE table=test-table [id=33cdf7b2e80b478cabe4ce18b5b26017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:41.698941 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d994812a1d841e8be1eb946654df000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:41.698941 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d994812a1d841e8be1eb946654df000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:41.699146 4616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d994812a1d841e8be1eb946654df000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:41.710165 4964 tablet_bootstrap.cc:492] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: Bootstrap starting.
I20260621 02:38:41.713222 4964 tablet_bootstrap.cc:654] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:41.714089 4966 tablet_bootstrap.cc:492] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: Bootstrap starting.
I20260621 02:38:41.714879 4965 tablet_bootstrap.cc:492] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: Bootstrap starting.
I20260621 02:38:41.715853 4964 log.cc:824] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:41.718190 4965 tablet_bootstrap.cc:654] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:41.718190 4966 tablet_bootstrap.cc:654] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:41.719465 4966 log.cc:824] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:41.719465 4965 log.cc:824] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:41.723170 4964 tablet_bootstrap.cc:492] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: No bootstrap required, opened a new log
I20260621 02:38:41.723425 4965 tablet_bootstrap.cc:492] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: No bootstrap required, opened a new log
I20260621 02:38:41.723448 4964 ts_tablet_manager.cc:1397] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20260621 02:38:41.723691 4965 ts_tablet_manager.cc:1397] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 02:38:41.724223 4966 tablet_bootstrap.cc:492] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: No bootstrap required, opened a new log
I20260621 02:38:41.724475 4966 ts_tablet_manager.cc:1397] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260621 02:38:41.730616 4964 raft_consensus.cc:348] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } }
I20260621 02:38:41.730975 4964 raft_consensus.cc:374] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:41.731077 4964 raft_consensus.cc:729] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c5a3a732a3488ab3d6b1a8174cbf3b, State: Initialized, Role: FOLLOWER
I20260621 02:38:41.731685 4964 consensus_queue.cc:260] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } }
I20260621 02:38:41.733465 4964 ts_tablet_manager.cc:1428] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260621 02:38:41.734021 4965 raft_consensus.cc:348] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } }
I20260621 02:38:41.734436 4965 raft_consensus.cc:374] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:41.734535 4965 raft_consensus.cc:729] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73117de690ad4d528fc8fbc508e94716, State: Initialized, Role: FOLLOWER
I20260621 02:38:41.735350 4965 consensus_queue.cc:260] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } }
I20260621 02:38:41.735838 4966 raft_consensus.cc:348] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } }
I20260621 02:38:41.736150 4966 raft_consensus.cc:374] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:41.736264 4966 raft_consensus.cc:729] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec10902c097453d96c11d6a7411362d, State: Initialized, Role: FOLLOWER
I20260621 02:38:41.736824 4966 consensus_queue.cc:260] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } }
I20260621 02:38:41.737371 4814 heartbeater.cc:503] Master 127.0.97.62:36147 was elected leader, sending a full tablet report...
I20260621 02:38:41.738039 4965 ts_tablet_manager.cc:1428] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260621 02:38:41.740167 4946 heartbeater.cc:503] Master 127.0.97.62:36147 was elected leader, sending a full tablet report...
I20260621 02:38:41.740641 4966 ts_tablet_manager.cc:1428] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: Time spent starting tablet: real 0.016s user 0.005s sys 0.010s
W20260621 02:38:41.837488 4947 tablet.cc:2369] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:38:41.869834 4815 tablet.cc:2369] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:38:41.870841 4682 tablet.cc:2369] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:38:41.878353 4972 raft_consensus.cc:482] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:41.878636 4972 raft_consensus.cc:504] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } }
I20260621 02:38:41.880411 4972 leader_election.cc:288] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73117de690ad4d528fc8fbc508e94716 (127.0.97.2:36899), 89c5a3a732a3488ab3d6b1a8174cbf3b (127.0.97.1:40061)
I20260621 02:38:41.887759 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.880861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45423 (local address 127.0.97.2:36899)
0621 02:38:41.881079 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.881088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.881112 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:38:41.881409 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.881415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.881494 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.881587 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:41.882040 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.883069 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.885518 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.886763 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.886899 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.887014 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:41.887329 (+ 315us) server_negotiation.cc:299] Negotiation successful
0621 02:38:41.887482 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 02:38:41.887773 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.880793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59607 (local address 127.0.97.1:40061)
0621 02:38:41.881102 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.881108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.881675 (+ 567us) server_negotiation.cc:407] Connection header received
0621 02:38:41.881806 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.881810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.881862 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.881929 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:41.882436 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.883370 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.885715 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.886908 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.887032 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.887119 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:41.887402 (+ 283us) server_negotiation.cc:299] Negotiation successful
0621 02:38:41.887517 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260621 02:38:41.887835 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.880707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36899 (local address 127.0.97.3:45423)
0621 02:38:41.881021 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:41.881046 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:41.881180 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:41.881637 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:41.881642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:41.881679 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:41.881879 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.881888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.883216 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.883221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.885361 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.885377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.886925 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.886929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.887157 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.887194 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:38:41.887299 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 02:38:41.887458 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":4,"spinlock_wait_cycles":896}
I20260621 02:38:41.887910 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.880829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40061 (local address 127.0.97.3:59607)
0621 02:38:41.881542 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:41.881579 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:41.881689 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:41.881974 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:41.881981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:41.882006 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:41.882270 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.882276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.883506 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.883511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.885580 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:41.885588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.887096 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.887101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:41.887250 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.887289 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:38:41.887363 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 02:38:41.887473 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":468,"threads_started":1}
I20260621 02:38:41.889233 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d994812a1d841e8be1eb946654df000" candidate_uuid: "8ec10902c097453d96c11d6a7411362d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73117de690ad4d528fc8fbc508e94716" is_pre_election: true
I20260621 02:38:41.889233 4636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d994812a1d841e8be1eb946654df000" candidate_uuid: "8ec10902c097453d96c11d6a7411362d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" is_pre_election: true
I20260621 02:38:41.889855 4636 raft_consensus.cc:2454] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ec10902c097453d96c11d6a7411362d in term 0.
I20260621 02:38:41.889855 4769 raft_consensus.cc:2454] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ec10902c097453d96c11d6a7411362d in term 0.
I20260621 02:38:41.890601 4837 leader_election.cc:302] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 73117de690ad4d528fc8fbc508e94716, 89c5a3a732a3488ab3d6b1a8174cbf3b, 8ec10902c097453d96c11d6a7411362d; no voters:
I20260621 02:38:41.891032 4972 raft_consensus.cc:2781] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:41.891180 4972 raft_consensus.cc:482] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:41.891286 4972 raft_consensus.cc:3037] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:41.893419 4972 raft_consensus.cc:504] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } }
I20260621 02:38:41.894009 4972 leader_election.cc:288] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [CANDIDATE]: Term 1 election: Requested vote from peers 73117de690ad4d528fc8fbc508e94716 (127.0.97.2:36899), 89c5a3a732a3488ab3d6b1a8174cbf3b (127.0.97.1:40061)
I20260621 02:38:41.894415 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d994812a1d841e8be1eb946654df000" candidate_uuid: "8ec10902c097453d96c11d6a7411362d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73117de690ad4d528fc8fbc508e94716"
I20260621 02:38:41.894415 4636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d994812a1d841e8be1eb946654df000" candidate_uuid: "8ec10902c097453d96c11d6a7411362d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b"
I20260621 02:38:41.894613 4769 raft_consensus.cc:3037] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:41.894616 4636 raft_consensus.cc:3037] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:41.897691 4769 raft_consensus.cc:2454] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ec10902c097453d96c11d6a7411362d in term 1.
I20260621 02:38:41.897689 4636 raft_consensus.cc:2454] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ec10902c097453d96c11d6a7411362d in term 1.
I20260621 02:38:41.898075 4836 leader_election.cc:302] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73117de690ad4d528fc8fbc508e94716, 8ec10902c097453d96c11d6a7411362d; no voters:
I20260621 02:38:41.898381 4972 raft_consensus.cc:2781] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:41.898813 4972 raft_consensus.cc:686] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 1 LEADER]: Becoming Leader. State: Replica: 8ec10902c097453d96c11d6a7411362d, State: Running, Role: LEADER
I20260621 02:38:41.899271 4972 consensus_queue.cc:237] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } }
I20260621 02:38:41.904124 4493 catalog_manager.cc:5495] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ec10902c097453d96c11d6a7411362d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8ec10902c097453d96c11d6a7411362d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ec10902c097453d96c11d6a7411362d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33615 } health_report { overall_health: HEALTHY } } }
I20260621 02:38:41.929726 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.924549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.0.97.3:33615)
0621 02:38:41.924766 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.924772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.924791 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:38:41.924945 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.924950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.925040 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.925186 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:41.925581 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.926298 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.927612 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.928005 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.928138 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:41.928594 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:41.929092 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:41.929248 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:41.929384 (+ 136us) server_negotiation.cc:299] Negotiation successful
0621 02:38:41.929460 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:38:41.937593 4636 raft_consensus.cc:1261] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [term 1 FOLLOWER]: Refusing update from remote peer 8ec10902c097453d96c11d6a7411362d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:41.937616 4769 raft_consensus.cc:1261] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [term 1 FOLLOWER]: Refusing update from remote peer 8ec10902c097453d96c11d6a7411362d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:41.938473 4974 consensus_queue.cc:1035] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c5a3a732a3488ab3d6b1a8174cbf3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:41.938798 4972 consensus_queue.cc:1035] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [LEADER]: Connected to new peer: Peer: permanent_uuid: "73117de690ad4d528fc8fbc508e94716" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:41.949477 4978 mvcc.cc:205] Tried to move back new op lower bound from 7299111001847177216 to 7299111001703231488. Current Snapshot: MvccSnapshot[applied={T|T < 7299111001847177216}]
I20260621 02:38:41.950493 4977 mvcc.cc:205] Tried to move back new op lower bound from 7299111001847177216 to 7299111001703231488. Current Snapshot: MvccSnapshot[applied={T|T < 7299111001847177216}]
I20260621 02:38:41.962106 4979 mvcc.cc:205] Tried to move back new op lower bound from 7299111001847177216 to 7299111001703231488. Current Snapshot: MvccSnapshot[applied={T|T < 7299111001847177216 or (T in {7299111001847177216})}]
I20260621 02:38:41.966694 4492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:41.966985 4492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:41.970705 4492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 909e228a2e23488e8f82089f8d2fbfc8: Sending DeleteTablet for 3 replicas of tablet 6d994812a1d841e8be1eb946654df000
I20260621 02:38:41.971750 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d994812a1d841e8be1eb946654df000 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39202
I20260621 02:38:41.971781 4616 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d994812a1d841e8be1eb946654df000 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42880
I20260621 02:38:41.971990 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d994812a1d841e8be1eb946654df000 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36330
I20260621 02:38:41.974629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 4554
I20260621 02:38:41.989091 4989 tablet_replica.cc:330] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: stopping tablet replica
I20260621 02:38:41.989748 4988 tablet_replica.cc:330] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: stopping tablet replica
I20260621 02:38:41.989964 4987 tablet_replica.cc:330] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: stopping tablet replica
I20260621 02:38:41.990346 4988 raft_consensus.cc:2229] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:41.990346 4987 raft_consensus.cc:2229] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:41.990821 4989 raft_consensus.cc:2229] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:38:41.990859 4987 raft_consensus.cc:2258] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:41.991444 4989 raft_consensus.cc:2258] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:41.992103 4988 raft_consensus.cc:2258] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:41.993521 4989 ts_tablet_manager.cc:1905] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:41.995146 4987 ts_tablet_manager.cc:1905] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:42.001194 4989 ts_tablet_manager.cc:1918] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:42.001482 4989 log.cc:1196] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/6d994812a1d841e8be1eb946654df000
Jun 21 02:38:41 dist-test-slave-jkn7 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:42.002027 4987 ts_tablet_manager.cc:1918] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:42.002209 4989 ts_tablet_manager.cc:1939] T 6d994812a1d841e8be1eb946654df000 P 8ec10902c097453d96c11d6a7411362d: Deleting consensus metadata
I20260621 02:38:42.002251 4987 log.cc:1196] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/6d994812a1d841e8be1eb946654df000
I20260621 02:38:42.002900 4987 ts_tablet_manager.cc:1939] T 6d994812a1d841e8be1eb946654df000 P 89c5a3a732a3488ab3d6b1a8174cbf3b: Deleting consensus metadata
I20260621 02:38:42.004331 4477 catalog_manager.cc:4841] TS 8ec10902c097453d96c11d6a7411362d (127.0.97.3:33615): tablet 6d994812a1d841e8be1eb946654df000 (table test-table [id=33cdf7b2e80b478cabe4ce18b5b26017]) successfully deleted
I20260621 02:38:42.004799 4480 catalog_manager.cc:4841] TS 89c5a3a732a3488ab3d6b1a8174cbf3b (127.0.97.1:40061): tablet 6d994812a1d841e8be1eb946654df000 (table test-table [id=33cdf7b2e80b478cabe4ce18b5b26017]) successfully deleted
I20260621 02:38:42.005810 4988 ts_tablet_manager.cc:1905] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:42.008994 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:41.975805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.0.97.1:40061)
0621 02:38:41.976003 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:41.976012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:41.991615 (+ 15603us) server_negotiation.cc:407] Connection header received
0621 02:38:41.991942 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:41.991949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:41.992034 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:41.992137 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:41.993926 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.995039 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.996850 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:41.997536 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:41.997645 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:42.002789 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:42.002805 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:42.002826 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:42.002863 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:42.006248 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:42.007087 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:42.007094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:42.007097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:42.007246 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:42.007618 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:42.007622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:42.007624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:42.007958 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:42.008179 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:42.008513 (+ 334us) server_negotiation.cc:299] Negotiation successful
0621 02:38:42.008668 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:38:42.009739 4676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:42.013599 4988 ts_tablet_manager.cc:1918] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:42.013862 4988 log.cc:1196] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/6d994812a1d841e8be1eb946654df000
I20260621 02:38:42.014621 4988 ts_tablet_manager.cc:1939] T 6d994812a1d841e8be1eb946654df000 P 73117de690ad4d528fc8fbc508e94716: Deleting consensus metadata
I20260621 02:38:42.017114 4479 catalog_manager.cc:4841] TS 73117de690ad4d528fc8fbc508e94716 (127.0.97.2:36899): tablet 6d994812a1d841e8be1eb946654df000 (table test-table [id=33cdf7b2e80b478cabe4ce18b5b26017]) successfully deleted
I20260621 02:38:42.127000 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 4554
I20260621 02:38:42.150804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 4687
Jun 21 02:38:42 dist-test-slave-jkn7 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:38:42.167568 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:42.151642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.0.97.2:36899)
0621 02:38:42.151834 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:42.151840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:42.151861 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:38:42.151969 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:42.151975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:42.152067 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:42.152192 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:42.153410 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:42.154548 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:42.155774 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:42.156389 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:42.156522 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:42.160981 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:42.161002 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:42.161023 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:42.161072 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:42.164813 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:42.165513 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:42.165524 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:42.165528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:42.165673 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:42.166019 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:42.166024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:42.166028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:42.166478 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:42.166758 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:42.166954 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 02:38:42.167177 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 02:38:42.168248 4809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:42.272396 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 4687
I20260621 02:38:42.289466 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 4819
Jun 21 02:38:42 dist-test-slave-jkn7 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:38:42.307111 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:42.290418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.0.97.3:33615)
0621 02:38:42.290600 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:42.290604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:42.290622 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:38:42.290750 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:42.290755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:42.290826 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:42.290907 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:42.292646 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:42.293547 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:42.295366 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:42.295795 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:42.295888 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:42.300924 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:42.300937 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:42.300952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:42.300986 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:42.304315 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:42.304990 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:42.304997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:42.304999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:42.305161 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:42.305562 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:42.305569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:42.305572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:42.305914 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:42.306230 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:42.306509 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 02:38:42.306725 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:38:42.307811 4941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:42.428670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 4819
I20260621 02:38:42.447788 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 4462
I20260621 02:38:42.448904 4523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:42.555744 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 4462
2026-06-21T02:38:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4210 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T02:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:42Z Disabled control of system clock
I20260621 02:38:42.621170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46085
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 02:38:42.932796 4998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:42.933096 4998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:42.933272 4998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:42.940382 4998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:42.940478 4998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:42.940562 4998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:42.940658 4998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:42.940734 4998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:42.949671 4998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:42.950902 4998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:42.952663 4998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:42.964151 5004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:42.964039 5006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:42.964671 5003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:42.964795 4998 server_base.cc:1034] running on GCE node
I20260621 02:38:42.965677 4998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:42.967067 4998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:42.968291 4998 hybrid_clock.cc:648] HybridClock initialized: now 1782009522968245 us; error 61 us; skew 500 ppm
I20260621 02:38:42.968628 4998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:38:42.970798 4998 webserver.cc:460] Webserver started at http://127.0.97.62:36171/ using document root <none> and password file <none>
I20260621 02:38:42.971477 4998 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:42.971608 4998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:42.971935 4998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:42.974726 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "d008f51f423a4542911eb66884254fba"
format_stamp: "Formatted at 2026-06-21 02:38:42 on dist-test-slave-jkn7"
server_key: "be9d14869dd59a67d0ba88b0fcbba11b"
server_key_iv: "1ac09bfa0aebcd000d8ce8ea3e6fb9bd"
server_key_version: "encryptionkey@0"
I20260621 02:38:42.975523 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "d008f51f423a4542911eb66884254fba"
format_stamp: "Formatted at 2026-06-21 02:38:42 on dist-test-slave-jkn7"
server_key: "be9d14869dd59a67d0ba88b0fcbba11b"
server_key_iv: "1ac09bfa0aebcd000d8ce8ea3e6fb9bd"
server_key_version: "encryptionkey@0"
I20260621 02:38:42.980436 4998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 02:38:42.983747 5012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:42.985024 4998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:42.985203 4998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "d008f51f423a4542911eb66884254fba"
format_stamp: "Formatted at 2026-06-21 02:38:42 on dist-test-slave-jkn7"
server_key: "be9d14869dd59a67d0ba88b0fcbba11b"
server_key_iv: "1ac09bfa0aebcd000d8ce8ea3e6fb9bd"
server_key_version: "encryptionkey@0"
I20260621 02:38:42.985479 4998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:42.990875 4998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:42.991606 4998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:42.991910 4998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:43.009480 4998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46085
I20260621 02:38:43.009452 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46085 every 8 connection(s)
I20260621 02:38:43.010742 4998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:43.014278 5064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:43.016971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 4998
I20260621 02:38:43.017163 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:43.017838 388 external_mini_cluster.cc:1428] Setting key 94b73eacb7ffb04dfa90a29ad6918b31
I20260621 02:38:43.026722 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:43.019774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.0.97.62:46085)
0621 02:38:43.020790 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:43.020814 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:43.020863 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:38:43.022743 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:43.022772 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:43.023319 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:43.023739 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:43.024719 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:43.024727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:43.024749 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:43.024769 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:43.024854 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:43.024877 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:43.025116 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:43.025248 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 02:38:43.025527 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":313,"threads_started":1}
I20260621 02:38:43.027259 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba: Bootstrap starting.
I20260621 02:38:43.030709 5064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:43.031669 5064 log.cc:824] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:43.034216 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba: No bootstrap required, opened a new log
I20260621 02:38:43.041875 5064 raft_consensus.cc:348] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d008f51f423a4542911eb66884254fba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46085 } }
I20260621 02:38:43.042235 5064 raft_consensus.cc:374] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:43.042347 5064 raft_consensus.cc:729] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d008f51f423a4542911eb66884254fba, State: Initialized, Role: FOLLOWER
I20260621 02:38:43.042886 5064 consensus_queue.cc:260] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d008f51f423a4542911eb66884254fba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46085 } }
I20260621 02:38:43.043118 5064 raft_consensus.cc:388] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:43.043260 5064 raft_consensus.cc:482] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:43.043432 5064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:43.045526 5064 raft_consensus.cc:504] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d008f51f423a4542911eb66884254fba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46085 } }
I20260621 02:38:43.045991 5064 leader_election.cc:302] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d008f51f423a4542911eb66884254fba; no voters:
I20260621 02:38:43.046604 5064 leader_election.cc:288] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:43.046841 5069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:43.047547 5069 raft_consensus.cc:686] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [term 1 LEADER]: Becoming Leader. State: Replica: d008f51f423a4542911eb66884254fba, State: Running, Role: LEADER
I20260621 02:38:43.048036 5069 consensus_queue.cc:237] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d008f51f423a4542911eb66884254fba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46085 } }
I20260621 02:38:43.048652 5064 sys_catalog.cc:564] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:43.050966 5071 sys_catalog.cc:455] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [sys.catalog]: SysCatalogTable state changed. Reason: New leader d008f51f423a4542911eb66884254fba. Latest consensus state: current_term: 1 leader_uuid: "d008f51f423a4542911eb66884254fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d008f51f423a4542911eb66884254fba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46085 } } }
I20260621 02:38:43.051365 5071 sys_catalog.cc:458] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:43.051405 5070 sys_catalog.cc:455] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d008f51f423a4542911eb66884254fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d008f51f423a4542911eb66884254fba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46085 } } }
I20260621 02:38:43.052767 5070 sys_catalog.cc:458] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:43.053508 5077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:43.057613 5077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:43.066131 5077 catalog_manager.cc:1347] Generated new cluster ID: 49bbeb455d2447c2854b9bedc46a569f
I20260621 02:38:43.066237 5077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:43.084774 5077 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:43.085253 5077 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:43.100112 5077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba: Generated new TSK 0
I20260621 02:38:43.101205 5077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:38:43.112247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46085
--builtin_ntp_servers=127.0.97.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 02:38:43.427255 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:43.427517 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:43.427680 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:43.434814 5088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:43.434906 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:43.435062 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:43.443674 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:43.444898 5088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:43.446632 5088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:43.459467 5093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:43.459513 5094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:43.459537 5096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:43.460589 5088 server_base.cc:1034] running on GCE node
I20260621 02:38:43.461146 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:43.462059 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:43.463256 5088 hybrid_clock.cc:648] HybridClock initialized: now 1782009523463211 us; error 36 us; skew 500 ppm
I20260621 02:38:43.463574 5088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:38:43.465706 5088 webserver.cc:460] Webserver started at http://127.0.97.1:45595/ using document root <none> and password file <none>
I20260621 02:38:43.466344 5088 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:43.466465 5088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:43.466768 5088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:43.469556 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "35c587f0625544509cdb7830a9a3a88f"
format_stamp: "Formatted at 2026-06-21 02:38:43 on dist-test-slave-jkn7"
server_key: "6713d75af92bd507d8d5afcc90c0f5d2"
server_key_iv: "cbc4f2c60a387583249e3598d83735d4"
server_key_version: "encryptionkey@0"
I20260621 02:38:43.470299 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "35c587f0625544509cdb7830a9a3a88f"
format_stamp: "Formatted at 2026-06-21 02:38:43 on dist-test-slave-jkn7"
server_key: "6713d75af92bd507d8d5afcc90c0f5d2"
server_key_iv: "cbc4f2c60a387583249e3598d83735d4"
server_key_version: "encryptionkey@0"
I20260621 02:38:43.475005 5088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:38:43.478189 5102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:43.479485 5088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:43.479655 5088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "35c587f0625544509cdb7830a9a3a88f"
format_stamp: "Formatted at 2026-06-21 02:38:43 on dist-test-slave-jkn7"
server_key: "6713d75af92bd507d8d5afcc90c0f5d2"
server_key_iv: "cbc4f2c60a387583249e3598d83735d4"
server_key_version: "encryptionkey@0"
I20260621 02:38:43.479871 5088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:43.485220 5088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:43.485991 5088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:43.486243 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:43.487099 5088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:43.488433 5088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:43.488546 5088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:43.488677 5088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:43.488762 5088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:43.520596 5088 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37211
I20260621 02:38:43.520645 5214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37211 every 8 connection(s)
I20260621 02:38:43.521952 5088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:43.529683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 5088
I20260621 02:38:43.529907 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:43.529898 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:43.524544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60093 (local address 127.0.97.62:46085)
0621 02:38:43.524730 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:43.524735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:43.526063 (+ 1328us) server_negotiation.cc:407] Connection header received
0621 02:38:43.527699 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:43.527705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:43.527773 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:43.527887 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:43.529149 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:43.529154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:43.529160 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:43.529176 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:43.529213 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:43.529224 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:43.529377 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:43.529568 (+ 191us) server_negotiation.cc:299] Negotiation successful
0621 02:38:43.529631 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:38:43.530540 388 external_mini_cluster.cc:1428] Setting key 4d39fd70d301ff2df2ff85e6baeadff8
I20260621 02:38:43.530776 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:43.524978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46085 (local address 127.0.97.1:60093)
0621 02:38:43.525918 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:43.525952 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:43.527438 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:43.528103 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:43.528111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:43.528194 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:43.528873 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:38:43.528876 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:43.528910 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:38:43.528987 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:43.529351 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:43.529360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:43.529368 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 02:38:43.529630 (+ 262us) client_negotiation.cc:240] Negotiation successful
0621 02:38:43.529861 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":339,"threads_started":1}
I20260621 02:38:43.532240 5215 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46085
I20260621 02:38:43.532517 5215 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:43.533102 5215 heartbeater.cc:511] Master 127.0.97.62:46085 requested a full tablet report, sending...
I20260621 02:38:43.533727 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46085
--builtin_ntp_servers=127.0.97.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 02:38:43.535207 5028 ts_manager.cc:194] Registered new tserver with Master: 35c587f0625544509cdb7830a9a3a88f (127.0.97.1:37211)
W20260621 02:38:43.847407 5218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:43.847697 5218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:43.847831 5218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:43.855052 5218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:43.855152 5218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:43.855259 5218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:43.864449 5218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:43.865729 5218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:43.867514 5218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:43.880995 5226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:43.881008 5223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:43.881018 5224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:43.882236 5218 server_base.cc:1034] running on GCE node
I20260621 02:38:43.882843 5218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:43.883854 5218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:43.885094 5218 hybrid_clock.cc:648] HybridClock initialized: now 1782009523885046 us; error 65 us; skew 500 ppm
I20260621 02:38:43.885494 5218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:38:43.887813 5218 webserver.cc:460] Webserver started at http://127.0.97.2:34341/ using document root <none> and password file <none>
I20260621 02:38:43.888619 5218 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:43.888746 5218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:43.889089 5218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:43.892095 5218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "93ab4eec9356491981a23367bf8d1c67"
format_stamp: "Formatted at 2026-06-21 02:38:43 on dist-test-slave-jkn7"
server_key: "9d8665d28e2bfd5e73241aa05ccac60b"
server_key_iv: "7a2a2c1849fa17f5b806b3205bda2c1b"
server_key_version: "encryptionkey@0"
I20260621 02:38:43.892889 5218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "93ab4eec9356491981a23367bf8d1c67"
format_stamp: "Formatted at 2026-06-21 02:38:43 on dist-test-slave-jkn7"
server_key: "9d8665d28e2bfd5e73241aa05ccac60b"
server_key_iv: "7a2a2c1849fa17f5b806b3205bda2c1b"
server_key_version: "encryptionkey@0"
I20260621 02:38:43.898028 5218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:38:43.901470 5232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:43.902796 5218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:38:43.902980 5218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "93ab4eec9356491981a23367bf8d1c67"
format_stamp: "Formatted at 2026-06-21 02:38:43 on dist-test-slave-jkn7"
server_key: "9d8665d28e2bfd5e73241aa05ccac60b"
server_key_iv: "7a2a2c1849fa17f5b806b3205bda2c1b"
server_key_version: "encryptionkey@0"
I20260621 02:38:43.903210 5218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:43.908823 5218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:43.909668 5218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:43.909940 5218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:43.910811 5218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:43.912166 5218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:43.912288 5218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:43.912428 5218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:43.912518 5218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:43.945232 5218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44483
I20260621 02:38:43.945328 5344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44483 every 8 connection(s)
I20260621 02:38:43.946583 5218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:43.952006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 5218
I20260621 02:38:43.952165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:43.952689 388 external_mini_cluster.cc:1428] Setting key b7ac4ff8a401d774590e308a76e0ec21
I20260621 02:38:43.955291 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:43.949157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47921 (local address 127.0.97.62:46085)
0621 02:38:43.949374 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:43.949379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:43.950818 (+ 1439us) server_negotiation.cc:407] Connection header received
0621 02:38:43.952930 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:43.952935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:43.952986 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:43.953071 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:43.954556 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:43.954561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:43.954565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:43.954583 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:43.954621 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:43.954635 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:43.954755 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:43.955036 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 02:38:43.955084 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:38:43.955819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46085
--builtin_ntp_servers=127.0.97.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 02:38:43.956519 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:43.949812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46085 (local address 127.0.97.2:47921)
0621 02:38:43.950668 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:43.950708 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:43.952640 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:43.953384 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:43.953392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:43.953481 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:43.954230 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:38:43.954232 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:43.954268 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:38:43.954357 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:43.954773 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:43.954786 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:43.954797 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:38:43.955087 (+ 290us) client_negotiation.cc:240] Negotiation successful
0621 02:38:43.955342 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":261,"threads_started":1}
I20260621 02:38:43.957887 5345 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46085
I20260621 02:38:43.958199 5345 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:43.958932 5345 heartbeater.cc:511] Master 127.0.97.62:46085 requested a full tablet report, sending...
I20260621 02:38:43.960541 5028 ts_manager.cc:194] Registered new tserver with Master: 93ab4eec9356491981a23367bf8d1c67 (127.0.97.2:44483)
W20260621 02:38:44.288200 5348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:44.288465 5348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:44.288606 5348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:44.295787 5348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:44.295914 5348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:44.296089 5348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:44.304747 5348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:44.305992 5348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:44.307669 5348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:44.320185 5353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:44.320247 5354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:44.320281 5356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:44.320953 5348 server_base.cc:1034] running on GCE node
I20260621 02:38:44.321543 5348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:44.322395 5348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:44.323592 5348 hybrid_clock.cc:648] HybridClock initialized: now 1782009524323540 us; error 47 us; skew 500 ppm
I20260621 02:38:44.323885 5348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:38:44.326082 5348 webserver.cc:460] Webserver started at http://127.0.97.3:36867/ using document root <none> and password file <none>
I20260621 02:38:44.326738 5348 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:44.326823 5348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:44.327167 5348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:44.329955 5348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "40784ace9b1c45d8a90b6d598aed6d9e"
format_stamp: "Formatted at 2026-06-21 02:38:44 on dist-test-slave-jkn7"
server_key: "65f9e1f2bf9b28db359f2651332372a1"
server_key_iv: "0f0c20aa999891df93a51ef7ff63352f"
server_key_version: "encryptionkey@0"
I20260621 02:38:44.330717 5348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "40784ace9b1c45d8a90b6d598aed6d9e"
format_stamp: "Formatted at 2026-06-21 02:38:44 on dist-test-slave-jkn7"
server_key: "65f9e1f2bf9b28db359f2651332372a1"
server_key_iv: "0f0c20aa999891df93a51ef7ff63352f"
server_key_version: "encryptionkey@0"
I20260621 02:38:44.335628 5348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:38:44.338986 5362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:44.340299 5348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:38:44.340492 5348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "40784ace9b1c45d8a90b6d598aed6d9e"
format_stamp: "Formatted at 2026-06-21 02:38:44 on dist-test-slave-jkn7"
server_key: "65f9e1f2bf9b28db359f2651332372a1"
server_key_iv: "0f0c20aa999891df93a51ef7ff63352f"
server_key_version: "encryptionkey@0"
I20260621 02:38:44.340718 5348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:44.346159 5348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:44.346869 5348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:44.347123 5348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:44.347971 5348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:44.349408 5348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:44.349520 5348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:44.349647 5348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:44.349742 5348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:44.381799 5348 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39739
I20260621 02:38:44.381881 5474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39739 every 8 connection(s)
I20260621 02:38:44.383080 5348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:44.385246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 5348
I20260621 02:38:44.385432 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:44.385991 388 external_mini_cluster.cc:1428] Setting key 4fd3cbd895b102f11fb50c7b1909588b
I20260621 02:38:44.391916 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.385874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42931 (local address 127.0.97.62:46085)
0621 02:38:44.386072 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.386078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.387781 (+ 1703us) server_negotiation.cc:407] Connection header received
0621 02:38:44.389758 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.389763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.389855 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.389985 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.391219 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.391225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.391230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.391247 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.391287 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.391298 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.391422 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.391600 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.391661 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:38:44.392740 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.386556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46085 (local address 127.0.97.3:42931)
0621 02:38:44.387632 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.387681 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:44.389526 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:44.390183 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:44.390190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:44.390273 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:44.390957 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:38:44.390959 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:44.390992 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:38:44.391070 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:44.391389 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:44.391399 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:44.391408 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:38:44.391677 (+ 269us) client_negotiation.cc:240] Negotiation successful
0621 02:38:44.391904 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":326,"threads_started":1}
I20260621 02:38:44.393992 5475 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46085
I20260621 02:38:44.394219 5475 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:44.394680 5475 heartbeater.cc:511] Master 127.0.97.62:46085 requested a full tablet report, sending...
I20260621 02:38:44.395879 5028 ts_manager.cc:194] Registered new tserver with Master: 40784ace9b1c45d8a90b6d598aed6d9e (127.0.97.3:39739)
I20260621 02:38:44.400976 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:38:44.407964 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.405147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.0.97.62:46085)
0621 02:38:44.405337 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.405342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.405984 (+ 642us) server_negotiation.cc:407] Connection header received
0621 02:38:44.406199 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.406204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.406259 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.406360 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.407346 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.407351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.407355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.407372 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.407410 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.407422 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.407540 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.407628 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.407702 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:38:44.411612 5028 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:38:44.418212 5028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:44.435212 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.429363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.0.97.2:44483)
0621 02:38:44.429955 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.429965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.430794 (+ 829us) server_negotiation.cc:407] Connection header received
0621 02:38:44.431072 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.431083 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.431308 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.431434 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.433976 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.433988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.434007 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.434039 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.434106 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.434129 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.434275 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.434583 (+ 308us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.434679 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":689,"threads_started":1}
I20260621 02:38:44.436457 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.429072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44483 (local address 127.0.0.1:57756)
0621 02:38:44.430606 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.430662 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:44.430917 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:44.432622 (+ 1705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:44.432631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:44.432728 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:44.433639 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:38:44.433645 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:44.433677 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:38:44.433764 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:44.434379 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:44.434389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:44.434396 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:38:44.435836 (+ 1440us) client_negotiation.cc:240] Negotiation successful
0621 02:38:44.435948 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"thread_start_us":5314,"threads_started":1}
I20260621 02:38:44.436971 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.429240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37211 (local address 127.0.0.1:41492)
0621 02:38:44.434617 (+ 5377us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.434633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:44.434715 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:44.435145 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:44.435149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:44.435170 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:44.435905 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:38:44.435911 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:44.435927 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:38:44.435971 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:44.436376 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:44.436381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:44.436386 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:38:44.436509 (+ 123us) client_negotiation.cc:240] Negotiation successful
0621 02:38:44.436559 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4220,"thread_start_us":879,"threads_started":1}
I20260621 02:38:44.438050 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.429881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.0.97.1:37211)
0621 02:38:44.434754 (+ 4873us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.434771 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.434798 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:38:44.434861 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.434866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.435004 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.435151 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.436099 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.436107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.436125 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.436153 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.436210 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.436233 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.437577 (+ 1344us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.437647 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.437737 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4728,"thread_start_us":4612,"threads_started":1}
I20260621 02:38:44.439642 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.431267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.0.97.3:39739)
0621 02:38:44.436525 (+ 5258us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.436534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.436563 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:38:44.436625 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.436632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.436821 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.437622 (+ 801us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.438838 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.438845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.438858 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.438888 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.438936 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.438954 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.439120 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.439180 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.439260 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5073,"thread_start_us":4918,"threads_started":1}
I20260621 02:38:44.439672 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.431024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39739 (local address 127.0.0.1:41710)
0621 02:38:44.434209 (+ 3185us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.434225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:44.434306 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:44.437017 (+ 2711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:44.437023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:44.437046 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:44.437791 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:38:44.437794 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:44.437809 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:38:44.437847 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:44.439068 (+ 1221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:44.439075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:44.439081 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:38:44.439198 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 02:38:44.439241 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3062,"thread_start_us":2499,"threads_started":1}
I20260621 02:38:44.440536 5280 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef06d4644244ac4a9239d5e68bf8cd1 (DEFAULT_TABLE table=test-table [id=c9a9b7ab69564ed2a4d5928934b66b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:44.441331 5150 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef06d4644244ac4a9239d5e68bf8cd1 (DEFAULT_TABLE table=test-table [id=c9a9b7ab69564ed2a4d5928934b66b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:44.442022 5280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef06d4644244ac4a9239d5e68bf8cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:44.442602 5150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef06d4644244ac4a9239d5e68bf8cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:44.442802 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef06d4644244ac4a9239d5e68bf8cd1 (DEFAULT_TABLE table=test-table [id=c9a9b7ab69564ed2a4d5928934b66b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:44.443930 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef06d4644244ac4a9239d5e68bf8cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:44.454391 5493 tablet_bootstrap.cc:492] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: Bootstrap starting.
I20260621 02:38:44.458206 5493 tablet_bootstrap.cc:654] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:44.459192 5493 log.cc:824] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:44.461556 5495 tablet_bootstrap.cc:492] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: Bootstrap starting.
I20260621 02:38:44.462121 5493 tablet_bootstrap.cc:492] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: No bootstrap required, opened a new log
I20260621 02:38:44.462371 5493 ts_tablet_manager.cc:1397] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:38:44.464354 5495 tablet_bootstrap.cc:654] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:44.465214 5495 log.cc:824] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:44.465432 5494 tablet_bootstrap.cc:492] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f: Bootstrap starting.
I20260621 02:38:44.467958 5495 tablet_bootstrap.cc:492] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: No bootstrap required, opened a new log
I20260621 02:38:44.468197 5495 ts_tablet_manager.cc:1397] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260621 02:38:44.468523 5494 tablet_bootstrap.cc:654] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:44.469655 5494 log.cc:824] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:44.469820 5493 raft_consensus.cc:348] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } }
I20260621 02:38:44.470285 5493 raft_consensus.cc:374] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:44.470397 5493 raft_consensus.cc:729] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40784ace9b1c45d8a90b6d598aed6d9e, State: Initialized, Role: FOLLOWER
I20260621 02:38:44.470983 5493 consensus_queue.cc:260] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } }
I20260621 02:38:44.472401 5494 tablet_bootstrap.cc:492] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f: No bootstrap required, opened a new log
I20260621 02:38:44.472637 5494 ts_tablet_manager.cc:1397] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260621 02:38:44.476833 5475 heartbeater.cc:503] Master 127.0.97.62:46085 was elected leader, sending a full tablet report...
I20260621 02:38:44.477358 5493 ts_tablet_manager.cc:1428] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260621 02:38:44.478369 5495 raft_consensus.cc:348] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } }
I20260621 02:38:44.478785 5495 raft_consensus.cc:374] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:44.478883 5495 raft_consensus.cc:729] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ab4eec9356491981a23367bf8d1c67, State: Initialized, Role: FOLLOWER
I20260621 02:38:44.479445 5495 consensus_queue.cc:260] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } }
I20260621 02:38:44.480525 5345 heartbeater.cc:503] Master 127.0.97.62:46085 was elected leader, sending a full tablet report...
I20260621 02:38:44.481146 5495 ts_tablet_manager.cc:1428] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260621 02:38:44.482676 5494 raft_consensus.cc:348] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } }
I20260621 02:38:44.483134 5494 raft_consensus.cc:374] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:44.483234 5494 raft_consensus.cc:729] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c587f0625544509cdb7830a9a3a88f, State: Initialized, Role: FOLLOWER
I20260621 02:38:44.483832 5494 consensus_queue.cc:260] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } }
I20260621 02:38:44.484884 5215 heartbeater.cc:503] Master 127.0.97.62:46085 was elected leader, sending a full tablet report...
I20260621 02:38:44.485409 5494 ts_tablet_manager.cc:1428] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260621 02:38:44.503155 5501 raft_consensus.cc:482] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:44.503491 5501 raft_consensus.cc:504] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } }
I20260621 02:38:44.505077 5501 leader_election.cc:288] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93ab4eec9356491981a23367bf8d1c67 (127.0.97.2:44483), 40784ace9b1c45d8a90b6d598aed6d9e (127.0.97.3:39739)
I20260621 02:38:44.508471 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.505379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47121 (local address 127.0.97.2:44483)
0621 02:38:44.505604 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.505613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.506434 (+ 821us) server_negotiation.cc:407] Connection header received
0621 02:38:44.506638 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.506651 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.506711 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.506788 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.507855 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.507861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.507866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.507884 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.507917 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.507931 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.508048 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.508126 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.508170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:38:44.508666 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.505658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50425 (local address 127.0.97.3:39739)
0621 02:38:44.505884 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.505893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.506479 (+ 586us) server_negotiation.cc:407] Connection header received
0621 02:38:44.506680 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.506683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.506727 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.506840 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.508058 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.508066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.508073 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.508096 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.508136 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.508155 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.508264 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.508369 (+ 105us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.508421 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:38:44.508684 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.505267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44483 (local address 127.0.97.1:47121)
0621 02:38:44.506312 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.506364 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:44.506480 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:44.506840 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:44.506845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:44.506868 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:44.507648 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:38:44.507651 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:44.507667 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:38:44.507711 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:44.508027 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:44.508031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:44.508036 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:38:44.508178 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 02:38:44.508232 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":729,"threads_started":1}
I20260621 02:38:44.508909 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.505375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39739 (local address 127.0.97.1:50425)
0621 02:38:44.506371 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.506392 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:44.506543 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:44.506880 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:44.506885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:44.506910 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:44.507855 (+ 945us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:38:44.507859 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:44.507873 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:38:44.507910 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:44.508254 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:44.508259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:44.508268 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:38:44.508409 (+ 141us) client_negotiation.cc:240] Negotiation successful
0621 02:38:44.508456 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":543,"threads_started":1}
I20260621 02:38:44.509815 5300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef06d4644244ac4a9239d5e68bf8cd1" candidate_uuid: "35c587f0625544509cdb7830a9a3a88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ab4eec9356491981a23367bf8d1c67" is_pre_election: true
I20260621 02:38:44.509814 5430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef06d4644244ac4a9239d5e68bf8cd1" candidate_uuid: "35c587f0625544509cdb7830a9a3a88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" is_pre_election: true
I20260621 02:38:44.510277 5430 raft_consensus.cc:2454] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c587f0625544509cdb7830a9a3a88f in term 0.
I20260621 02:38:44.510282 5300 raft_consensus.cc:2454] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c587f0625544509cdb7830a9a3a88f in term 0.
I20260621 02:38:44.510946 5104 leader_election.cc:302] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35c587f0625544509cdb7830a9a3a88f, 40784ace9b1c45d8a90b6d598aed6d9e, 93ab4eec9356491981a23367bf8d1c67; no voters:
I20260621 02:38:44.511406 5501 raft_consensus.cc:2781] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:44.511547 5501 raft_consensus.cc:482] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:44.511658 5501 raft_consensus.cc:3037] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:44.513767 5501 raft_consensus.cc:504] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } }
I20260621 02:38:44.514343 5501 leader_election.cc:288] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [CANDIDATE]: Term 1 election: Requested vote from peers 93ab4eec9356491981a23367bf8d1c67 (127.0.97.2:44483), 40784ace9b1c45d8a90b6d598aed6d9e (127.0.97.3:39739)
I20260621 02:38:44.514744 5300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef06d4644244ac4a9239d5e68bf8cd1" candidate_uuid: "35c587f0625544509cdb7830a9a3a88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ab4eec9356491981a23367bf8d1c67"
I20260621 02:38:44.514801 5430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef06d4644244ac4a9239d5e68bf8cd1" candidate_uuid: "35c587f0625544509cdb7830a9a3a88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40784ace9b1c45d8a90b6d598aed6d9e"
I20260621 02:38:44.514947 5300 raft_consensus.cc:3037] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:44.514983 5430 raft_consensus.cc:3037] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:44.517035 5300 raft_consensus.cc:2454] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c587f0625544509cdb7830a9a3a88f in term 1.
I20260621 02:38:44.517052 5430 raft_consensus.cc:2454] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c587f0625544509cdb7830a9a3a88f in term 1.
I20260621 02:38:44.517496 5106 leader_election.cc:302] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35c587f0625544509cdb7830a9a3a88f, 40784ace9b1c45d8a90b6d598aed6d9e, 93ab4eec9356491981a23367bf8d1c67; no voters:
I20260621 02:38:44.517846 5501 raft_consensus.cc:2781] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:44.518265 5501 raft_consensus.cc:686] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 1 LEADER]: Becoming Leader. State: Replica: 35c587f0625544509cdb7830a9a3a88f, State: Running, Role: LEADER
I20260621 02:38:44.518690 5501 consensus_queue.cc:237] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } }
I20260621 02:38:44.523101 5029 catalog_manager.cc:5495] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f reported cstate change: term changed from 0 to 1, leader changed from <none> to 35c587f0625544509cdb7830a9a3a88f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "35c587f0625544509cdb7830a9a3a88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35c587f0625544509cdb7830a9a3a88f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 } health_report { overall_health: UNKNOWN } } }
W20260621 02:38:44.524077 5216 tablet.cc:2369] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:38:44.538524 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.536789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.0.97.1:37211)
0621 02:38:44.536938 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.536942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.536957 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:38:44.537021 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.537024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.537086 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.537162 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.538057 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.538060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.538063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.538075 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.538100 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.538110 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.538192 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.538304 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.538342 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:38:44.546000 5300 raft_consensus.cc:1261] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [term 1 FOLLOWER]: Refusing update from remote peer 35c587f0625544509cdb7830a9a3a88f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:44.546005 5430 raft_consensus.cc:1261] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [term 1 FOLLOWER]: Refusing update from remote peer 35c587f0625544509cdb7830a9a3a88f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:44.546847 5501 consensus_queue.cc:1035] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ab4eec9356491981a23367bf8d1c67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:44.547135 5504 consensus_queue.cc:1035] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40784ace9b1c45d8a90b6d598aed6d9e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:44.570473 5029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:44.570761 5029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:44.574198 5029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d008f51f423a4542911eb66884254fba: Sending DeleteTablet for 3 replicas of tablet 9ef06d4644244ac4a9239d5e68bf8cd1
I20260621 02:38:44.575407 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef06d4644244ac4a9239d5e68bf8cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:44 UTC) from {username='slave'} at 127.0.0.1:41710
I20260621 02:38:44.575397 5280 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef06d4644244ac4a9239d5e68bf8cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:44 UTC) from {username='slave'} at 127.0.0.1:57756
I20260621 02:38:44.575531 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef06d4644244ac4a9239d5e68bf8cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:44 UTC) from {username='slave'} at 127.0.0.1:41492
I20260621 02:38:44.577375 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 5088
I20260621 02:38:44.586652 5518 tablet_replica.cc:330] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: stopping tablet replica
I20260621 02:38:44.587311 5519 tablet_replica.cc:330] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f: stopping tablet replica
I20260621 02:38:44.587996 5519 raft_consensus.cc:2229] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:38:44.588572 5518 raft_consensus.cc:2229] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:44.589241 5518 raft_consensus.cc:2258] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:44.590044 5519 raft_consensus.cc:2258] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 35c587f0625544509cdb7830a9a3a88f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:44.592888 5518 ts_tablet_manager.cc:1905] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:44.595100 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.578879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.0.97.1:37211)
0621 02:38:44.579045 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.579051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.592161 (+ 13110us) server_negotiation.cc:407] Connection header received
0621 02:38:44.592620 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.592626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.592685 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.592756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.594364 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.594370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.594375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.594394 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.594428 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.594443 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.594534 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.594762 (+ 228us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.594820 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:38:44.596329 5210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:44.597167 5517 tablet_replica.cc:330] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: stopping tablet replica
I20260621 02:38:44.597843 5517 raft_consensus.cc:2229] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:44.598544 5517 raft_consensus.cc:2258] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:44.600935 5517 ts_tablet_manager.cc:1905] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:44.601995 5518 ts_tablet_manager.cc:1918] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:44.602170 5518 log.cc:1196] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/9ef06d4644244ac4a9239d5e68bf8cd1
I20260621 02:38:44.602604 5518 ts_tablet_manager.cc:1939] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 40784ace9b1c45d8a90b6d598aed6d9e: Deleting consensus metadata
I20260621 02:38:44.604085 5014 catalog_manager.cc:4841] TS 40784ace9b1c45d8a90b6d598aed6d9e (127.0.97.3:39739): tablet 9ef06d4644244ac4a9239d5e68bf8cd1 (table test-table [id=c9a9b7ab69564ed2a4d5928934b66b6c]) successfully deleted
I20260621 02:38:44.608137 5517 ts_tablet_manager.cc:1918] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:44.608325 5517 log.cc:1196] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/9ef06d4644244ac4a9239d5e68bf8cd1
I20260621 02:38:44.608860 5517 ts_tablet_manager.cc:1939] T 9ef06d4644244ac4a9239d5e68bf8cd1 P 93ab4eec9356491981a23367bf8d1c67: Deleting consensus metadata
I20260621 02:38:44.610747 5016 catalog_manager.cc:4841] TS 93ab4eec9356491981a23367bf8d1c67 (127.0.97.2:44483): tablet 9ef06d4644244ac4a9239d5e68bf8cd1 (table test-table [id=c9a9b7ab69564ed2a4d5928934b66b6c]) successfully deleted
I20260621 02:38:44.691701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 5088
I20260621 02:38:44.708925 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 5218
W20260621 02:38:44.709142 5014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37211 (error 108)
I20260621 02:38:44.711033 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.710551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37211 (local address 127.0.0.1:41516)
0621 02:38:44.710713 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.710866 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:38:44.711272 5014 catalog_manager.cc:4584] TS 35c587f0625544509cdb7830a9a3a88f (127.0.97.1:37211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ef06d4644244ac4a9239d5e68bf8cd1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37211: connect: Connection refused (error 111)
I20260621 02:38:44.712095 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.709752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.0.97.2:44483)
0621 02:38:44.710124 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.710128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.710146 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:38:44.710287 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.710291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.710384 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.710507 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.711472 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.711477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.711481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.711497 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.711529 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.711541 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.711679 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.711785 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.711844 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260621 02:38:44.712709 5340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:44.764734 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.764185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37211 (local address 127.0.0.1:41522)
0621 02:38:44.764418 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.764537 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 02:38:44.765799 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.765448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37211 (local address 127.0.0.1:41528)
0621 02:38:44.765592 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.765671 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 02:38:44.800009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 5218
I20260621 02:38:44.815828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 5348
I20260621 02:38:44.818521 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.816557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.0.97.3:39739)
0621 02:38:44.816714 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:44.816719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:44.816752 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:38:44.816913 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:44.816917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:44.817006 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:44.817091 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:44.818008 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:44.818013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:44.818016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:44.818031 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:44.818060 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:44.818071 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:44.818175 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:44.818231 (+ 56us) server_negotiation.cc:299] Negotiation successful
0621 02:38:44.818275 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:38:44.819131 5470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:44.826066 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.825668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37211 (local address 127.0.0.1:41540)
0621 02:38:44.825837 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.825935 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 02:38:44.826969 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.826646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37211 (local address 127.0.0.1:41542)
0621 02:38:44.826787 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.826860 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 02:38:44.905040 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 5348
I20260621 02:38:44.907554 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.907111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37211 (local address 127.0.0.1:41554)
0621 02:38:44.907330 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.907410 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 02:38:44.908592 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:44.908166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37211 (local address 127.0.0.1:41562)
0621 02:38:44.908420 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:44.908481 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260621 02:38:44.921468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 4998
I20260621 02:38:44.922502 5059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:45.026927 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 4998
2026-06-21T02:38:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2466 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 02:38:45.063683 388 mini_oidc.cc:152] Starting JWKS server
I20260621 02:38:45.066213 388 webserver.cc:460] Webserver started at https://127.0.0.1:37019/ using document root <none> and password file <none>
I20260621 02:38:45.066524 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 02:38:45.067107 388 webserver.cc:460] Webserver started at http://127.0.0.1:35251/ using document root <none> and password file <none>
2026-06-21T02:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:45Z Disabled control of system clock
I20260621 02:38:45.092281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37875
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37875
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37019/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/testchainca.cert with env {}
W20260621 02:38:45.405433 5531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:45.405711 5531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:45.405879 5531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:45.412797 5531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:45.412889 5531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:45.412969 5531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:45.413054 5531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:45.413156 5531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:38:45.413259 5531 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:38:45.413345 5531 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37019/jwks.json
I20260621 02:38:45.423070 5531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37019/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:45.424328 5531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:45.425654 5531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:45.436700 5537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:45.436858 5536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:45.436826 5539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:45.437883 5531 server_base.cc:1034] running on GCE node
I20260621 02:38:45.438657 5531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:45.439963 5531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:45.441154 5531 hybrid_clock.cc:648] HybridClock initialized: now 1782009525441122 us; error 25 us; skew 500 ppm
I20260621 02:38:45.441533 5531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:38:45.443727 5531 webserver.cc:460] Webserver started at http://127.0.97.62:38499/ using document root <none> and password file <none>
I20260621 02:38:45.444389 5531 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:45.444509 5531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:45.444821 5531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:45.447590 5531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "524bc47fdc1b468ba79578afdf1a6bae"
format_stamp: "Formatted at 2026-06-21 02:38:45 on dist-test-slave-jkn7"
server_key: "f9fd1938580171d159a04b88f3cb6abe"
server_key_iv: "988a629b836cf24ed107e0bf43771e51"
server_key_version: "encryptionkey@0"
I20260621 02:38:45.448385 5531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "524bc47fdc1b468ba79578afdf1a6bae"
format_stamp: "Formatted at 2026-06-21 02:38:45 on dist-test-slave-jkn7"
server_key: "f9fd1938580171d159a04b88f3cb6abe"
server_key_iv: "988a629b836cf24ed107e0bf43771e51"
server_key_version: "encryptionkey@0"
I20260621 02:38:45.453145 5531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260621 02:38:45.456271 5545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:45.457603 5531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:38:45.457782 5531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "524bc47fdc1b468ba79578afdf1a6bae"
format_stamp: "Formatted at 2026-06-21 02:38:45 on dist-test-slave-jkn7"
server_key: "f9fd1938580171d159a04b88f3cb6abe"
server_key_iv: "988a629b836cf24ed107e0bf43771e51"
server_key_version: "encryptionkey@0"
I20260621 02:38:45.457983 5531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:45.480434 5531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:45.481223 5531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:45.481590 5531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:45.499521 5531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37875
I20260621 02:38:45.499539 5596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37875 every 8 connection(s)
I20260621 02:38:45.500813 5531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:45.504364 5597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:45.508584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 5531
I20260621 02:38:45.508795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:45.509614 388 external_mini_cluster.cc:1428] Setting key d3d73312722b5bfb738a61a2d9e14094
I20260621 02:38:45.515275 5597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae: Bootstrap starting.
I20260621 02:38:45.518216 5597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:45.519240 5597 log.cc:824] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:45.522311 5597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae: No bootstrap required, opened a new log
I20260621 02:38:45.522744 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:45.511513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.0.97.62:37875)
0621 02:38:45.512408 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:45.512417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:45.512457 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:38:45.514208 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:45.514229 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:45.514646 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:45.515016 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:45.515561 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:45.516818 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:45.518202 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:45.518913 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:45.519128 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:45.520354 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:45.520367 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:45.520388 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:45.520989 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:45.521057 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:45.521077 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:45.521163 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:45.521493 (+ 330us) server_negotiation.cc:299] Negotiation successful
0621 02:38:45.521806 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":294,"threads_started":1}
I20260621 02:38:45.526295 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:38:45.530972 5597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524bc47fdc1b468ba79578afdf1a6bae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37875 } }
I20260621 02:38:45.531303 5597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:45.531375 5597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524bc47fdc1b468ba79578afdf1a6bae, State: Initialized, Role: FOLLOWER
I20260621 02:38:45.531843 5597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524bc47fdc1b468ba79578afdf1a6bae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37875 } }
I20260621 02:38:45.532027 5597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:45.532117 5597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:45.532238 5597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:45.534390 5597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524bc47fdc1b468ba79578afdf1a6bae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37875 } }
I20260621 02:38:45.534921 5597 leader_election.cc:302] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 524bc47fdc1b468ba79578afdf1a6bae; no voters:
I20260621 02:38:45.535583 5597 leader_election.cc:288] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:45.535697 5602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:45.536374 5602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [term 1 LEADER]: Becoming Leader. State: Replica: 524bc47fdc1b468ba79578afdf1a6bae, State: Running, Role: LEADER
I20260621 02:38:45.536970 5602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524bc47fdc1b468ba79578afdf1a6bae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37875 } }
I20260621 02:38:45.537880 5597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:45.539978 5603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "524bc47fdc1b468ba79578afdf1a6bae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524bc47fdc1b468ba79578afdf1a6bae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37875 } } }
I20260621 02:38:45.540328 5603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:45.540174 5604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 524bc47fdc1b468ba79578afdf1a6bae. Latest consensus state: current_term: 1 leader_uuid: "524bc47fdc1b468ba79578afdf1a6bae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524bc47fdc1b468ba79578afdf1a6bae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37875 } } }
I20260621 02:38:45.540589 5604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:45.543293 5609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:45.548530 5609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:45.557086 5609 catalog_manager.cc:1347] Generated new cluster ID: c7612599ae114810be1a6c6eed15f6cd
I20260621 02:38:45.557206 5609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:45.566111 5609 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:45.567909 5609 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:45.582159 5609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 524bc47fdc1b468ba79578afdf1a6bae: Generated new TSK 0
I20260621 02:38:45.583251 5609 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 02:38:45.623314 5627 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 02:38:45.623507 5627 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 02:38:45.624454 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:45.603445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.0.97.62:37875)
0621 02:38:45.603687 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:45.603696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:45.604159 (+ 463us) server_negotiation.cc:407] Connection header received
0621 02:38:45.604489 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:45.604492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:45.604559 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:45.604675 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:38:45.605374 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:45.606090 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:45.608136 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:45.608606 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:45.608710 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:45.609359 (+ 649us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:38:45.623919 (+ 14560us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 02:38:45.624224 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 02:38:45.624596 5600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260621 02:38:45.626863 5626 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:38:45.603394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37875 (local address 127.0.0.1:48456)
0621 02:38:45.604048 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:45.604069 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:45.604261 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:45.604732 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:45.604738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:45.604751 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:38:45.605132 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:45.605144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:45.606222 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:45.606227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:45.607991 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:45.608006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:45.608733 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:45.608779 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:45.608984 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:45.609052 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:38:45.624238 (+ 15186us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 02:38:45.624607 (+ 369us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37875: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":339,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:37875: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260621 02:38:45.629007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 5531
I20260621 02:38:45.630208 5592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:45.721468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 5531
2026-06-21T02:38:45Z chronyd exiting
I20260621 02:38:45.771790 388 test_util.cc:175] -----------------------------------------------
I20260621 02:38:45.771930 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009483336832-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (712 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 02:38:45.775609 388 mini_oidc.cc:152] Starting JWKS server
I20260621 02:38:45.777457 388 webserver.cc:460] Webserver started at https://127.0.0.1:34325/ using document root <none> and password file <none>
I20260621 02:38:45.777678 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 02:38:45.778316 388 webserver.cc:460] Webserver started at http://127.0.0.1:44029/ using document root <none> and password file <none>
2026-06-21T02:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:45Z Disabled control of system clock
I20260621 02:38:45.802834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37641
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37641
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34325/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/testchainca.cert with env {}
W20260621 02:38:46.115938 5634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:46.116269 5634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:46.116456 5634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:46.123476 5634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:46.123610 5634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:46.123699 5634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:46.123805 5634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:46.123890 5634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:38:46.123977 5634 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:38:46.124056 5634 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34325/jwks.json
I20260621 02:38:46.133410 5634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34325/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:46.134694 5634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:46.136018 5634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:46.147739 5642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:46.147763 5640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:46.147794 5639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:46.148430 5634 server_base.cc:1034] running on GCE node
I20260621 02:38:46.149215 5634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:46.150585 5634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:46.151814 5634 hybrid_clock.cc:648] HybridClock initialized: now 1782009526151785 us; error 45 us; skew 500 ppm
I20260621 02:38:46.152156 5634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:38:46.154624 5634 webserver.cc:460] Webserver started at http://127.0.97.62:40011/ using document root <none> and password file <none>
I20260621 02:38:46.155298 5634 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:46.155431 5634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:46.155755 5634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:46.158535 5634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "91fb436d695b4e8b971910d68025188c"
format_stamp: "Formatted at 2026-06-21 02:38:46 on dist-test-slave-jkn7"
server_key: "3369afb7384c9242b02231c010c5bc56"
server_key_iv: "5173e7bcac65598cc1bc4228609b224d"
server_key_version: "encryptionkey@0"
I20260621 02:38:46.159401 5634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "91fb436d695b4e8b971910d68025188c"
format_stamp: "Formatted at 2026-06-21 02:38:46 on dist-test-slave-jkn7"
server_key: "3369afb7384c9242b02231c010c5bc56"
server_key_iv: "5173e7bcac65598cc1bc4228609b224d"
server_key_version: "encryptionkey@0"
I20260621 02:38:46.164361 5634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:38:46.167718 5648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:46.169015 5634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:46.169190 5634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "91fb436d695b4e8b971910d68025188c"
format_stamp: "Formatted at 2026-06-21 02:38:46 on dist-test-slave-jkn7"
server_key: "3369afb7384c9242b02231c010c5bc56"
server_key_iv: "5173e7bcac65598cc1bc4228609b224d"
server_key_version: "encryptionkey@0"
I20260621 02:38:46.169442 5634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:46.187513 5634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:46.188186 5634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:46.188418 5634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:46.205806 5634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37641
I20260621 02:38:46.205842 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37641 every 8 connection(s)
I20260621 02:38:46.207046 5634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:46.210425 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 5634
I20260621 02:38:46.210613 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:46.210706 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:46.211397 388 external_mini_cluster.cc:1428] Setting key 1943859d1266b8689a081bea3aef967c
I20260621 02:38:46.222118 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c: Bootstrap starting.
I20260621 02:38:46.224364 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:46.213317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.0.97.62:37641)
0621 02:38:46.214315 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:46.214325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:46.214365 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:38:46.216506 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:46.216537 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:46.216959 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:46.217379 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:46.217869 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:46.219055 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:46.220323 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:46.220948 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:46.221145 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:46.222348 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:46.222356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:46.222369 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:46.222851 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:46.222920 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:46.222943 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:46.223062 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:46.223268 (+ 206us) server_negotiation.cc:299] Negotiation successful
0621 02:38:46.223541 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":353,"threads_started":1}
I20260621 02:38:46.225275 5700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:46.226302 5700 log.cc:824] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:46.227862 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:38:46.228965 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c: No bootstrap required, opened a new log
I20260621 02:38:46.236055 5700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fb436d695b4e8b971910d68025188c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37641 } }
I20260621 02:38:46.236374 5700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:46.236474 5700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fb436d695b4e8b971910d68025188c, State: Initialized, Role: FOLLOWER
I20260621 02:38:46.237046 5700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fb436d695b4e8b971910d68025188c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37641 } }
I20260621 02:38:46.237233 5700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:46.237404 5700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:46.237591 5700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:46.239681 5700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fb436d695b4e8b971910d68025188c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37641 } }
I20260621 02:38:46.240144 5700 leader_election.cc:302] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91fb436d695b4e8b971910d68025188c; no voters:
I20260621 02:38:46.240658 5700 leader_election.cc:288] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:46.240813 5705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:46.241525 5705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [term 1 LEADER]: Becoming Leader. State: Replica: 91fb436d695b4e8b971910d68025188c, State: Running, Role: LEADER
I20260621 02:38:46.242043 5705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fb436d695b4e8b971910d68025188c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37641 } }
I20260621 02:38:46.242689 5700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:46.245133 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91fb436d695b4e8b971910d68025188c. Latest consensus state: current_term: 1 leader_uuid: "91fb436d695b4e8b971910d68025188c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fb436d695b4e8b971910d68025188c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37641 } } }
I20260621 02:38:46.245625 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:46.245631 5706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91fb436d695b4e8b971910d68025188c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fb436d695b4e8b971910d68025188c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37641 } } }
I20260621 02:38:46.246376 5706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:46.246984 5715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:46.251552 5715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:46.258944 5715 catalog_manager.cc:1347] Generated new cluster ID: 9414fd878dc5421291db711c992817f3
I20260621 02:38:46.259054 5715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:46.271934 5715 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:46.273660 5715 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:46.290856 5715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91fb436d695b4e8b971910d68025188c: Generated new TSK 0
I20260621 02:38:46.291635 5715 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 02:38:46.331566 5730 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 02:38:46.331696 5730 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 02:38:46.332667 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:46.313220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.0.97.62:37641)
0621 02:38:46.313455 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:46.313464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:46.313991 (+ 527us) server_negotiation.cc:407] Connection header received
0621 02:38:46.314220 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:46.314224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:46.314286 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:46.314398 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:38:46.314940 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:46.315632 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:46.317663 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:46.318087 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:46.318194 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:46.318742 (+ 548us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:38:46.332169 (+ 13427us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 02:38:46.332445 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":85}
W20260621 02:38:46.332808 5703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260621 02:38:46.333039 5729 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:38:46.313187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37641 (local address 127.0.0.1:51574)
0621 02:38:46.313874 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:46.313893 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:46.314039 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:46.314448 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:46.314453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:46.314466 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:38:46.314787 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:46.314799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:46.315755 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:46.315760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:46.317530 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:46.317544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:46.318204 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:46.318270 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:46.318470 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:46.318513 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:38:46.332385 (+ 13872us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 02:38:46.332651 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37641: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":383,"threads_started":1}
I20260621 02:38:46.334317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 5634
I20260621 02:38:46.335356 5695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:46.426613 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 5634
2026-06-21T02:38:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (705 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 02:38:46.480755 388 mini_oidc.cc:152] Starting JWKS server
I20260621 02:38:46.482609 388 webserver.cc:460] Webserver started at https://127.0.0.1:33541/ using document root <none> and password file <none>
I20260621 02:38:46.482880 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 02:38:46.483510 388 webserver.cc:460] Webserver started at http://127.0.0.1:39001/ using document root <none> and password file <none>
2026-06-21T02:38:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:46Z Disabled control of system clock
I20260621 02:38:46.508121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40941
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33541/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 02:38:46.814332 5737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:46.814700 5737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:46.814879 5737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:46.822151 5737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:46.822333 5737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:46.822435 5737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:46.822541 5737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:46.822628 5737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:38:46.822721 5737 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:38:46.822781 5737 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33541/jwks.json
I20260621 02:38:46.832074 5737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33541/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:46.833531 5737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:46.835356 5737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:46.847734 5742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:46.847762 5745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:46.847819 5743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:46.848915 5737 server_base.cc:1034] running on GCE node
I20260621 02:38:46.849797 5737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:46.851178 5737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:46.852412 5737 hybrid_clock.cc:648] HybridClock initialized: now 1782009526852374 us; error 53 us; skew 500 ppm
I20260621 02:38:46.852756 5737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:38:46.858492 5737 webserver.cc:460] Webserver started at http://127.0.97.62:33861/ using document root <none> and password file <none>
I20260621 02:38:46.859223 5737 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:46.859370 5737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:46.859712 5737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:46.862722 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "c20bcd3b33f34c7bba590200ff5dae37"
format_stamp: "Formatted at 2026-06-21 02:38:46 on dist-test-slave-jkn7"
server_key: "ac6dd9746377ea7633328ad1e3abe75f"
server_key_iv: "941b79279466421dcb06562d4ed2545e"
server_key_version: "encryptionkey@0"
I20260621 02:38:46.863552 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "c20bcd3b33f34c7bba590200ff5dae37"
format_stamp: "Formatted at 2026-06-21 02:38:46 on dist-test-slave-jkn7"
server_key: "ac6dd9746377ea7633328ad1e3abe75f"
server_key_iv: "941b79279466421dcb06562d4ed2545e"
server_key_version: "encryptionkey@0"
I20260621 02:38:46.868904 5737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 02:38:46.872457 5751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:46.873970 5737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:46.874185 5737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "c20bcd3b33f34c7bba590200ff5dae37"
format_stamp: "Formatted at 2026-06-21 02:38:46 on dist-test-slave-jkn7"
server_key: "ac6dd9746377ea7633328ad1e3abe75f"
server_key_iv: "941b79279466421dcb06562d4ed2545e"
server_key_version: "encryptionkey@0"
I20260621 02:38:46.874431 5737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:46.886863 5737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:46.887676 5737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:46.887956 5737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:46.905802 5737 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40941
I20260621 02:38:46.905820 5802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40941 every 8 connection(s)
I20260621 02:38:46.907171 5737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:46.910881 5803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:46.913856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 5737
I20260621 02:38:46.914081 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:46.914901 388 external_mini_cluster.cc:1428] Setting key 8647f35e495dc05c1918a0fbc981cd75
I20260621 02:38:46.922938 5803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37: Bootstrap starting.
I20260621 02:38:46.926091 5803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:46.927215 5803 log.cc:824] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:46.928525 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:46.916934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.0.97.62:40941)
0621 02:38:46.917943 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:46.917955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:46.918008 (+ 53us) server_negotiation.cc:407] Connection header received
0621 02:38:46.919586 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:46.919608 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:46.920116 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:46.920562 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:46.921148 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:46.922447 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:46.923780 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:46.924457 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:46.924654 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:46.925833 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:46.925840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:46.925854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:46.926428 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:46.926528 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:46.926547 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:46.926673 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:46.926892 (+ 219us) server_negotiation.cc:299] Negotiation successful
0621 02:38:46.927181 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":369,"threads_started":1}
I20260621 02:38:46.930464 5803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37: No bootstrap required, opened a new log
I20260621 02:38:46.932103 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:38:46.938087 5803 raft_consensus.cc:348] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bcd3b33f34c7bba590200ff5dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } }
I20260621 02:38:46.938453 5803 raft_consensus.cc:374] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:46.938560 5803 raft_consensus.cc:729] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20bcd3b33f34c7bba590200ff5dae37, State: Initialized, Role: FOLLOWER
I20260621 02:38:46.939086 5803 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bcd3b33f34c7bba590200ff5dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } }
I20260621 02:38:46.939342 5803 raft_consensus.cc:388] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:46.939481 5803 raft_consensus.cc:482] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:46.939646 5803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:46.942044 5803 raft_consensus.cc:504] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bcd3b33f34c7bba590200ff5dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } }
I20260621 02:38:46.942526 5803 leader_election.cc:302] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c20bcd3b33f34c7bba590200ff5dae37; no voters:
I20260621 02:38:46.943087 5803 leader_election.cc:288] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:46.943212 5808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:46.944365 5808 raft_consensus.cc:686] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [term 1 LEADER]: Becoming Leader. State: Replica: c20bcd3b33f34c7bba590200ff5dae37, State: Running, Role: LEADER
I20260621 02:38:46.944895 5808 consensus_queue.cc:237] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bcd3b33f34c7bba590200ff5dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } }
I20260621 02:38:46.945338 5803 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:46.947590 5809 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c20bcd3b33f34c7bba590200ff5dae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bcd3b33f34c7bba590200ff5dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } } }
I20260621 02:38:46.948056 5809 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:46.948894 5817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:46.948849 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c20bcd3b33f34c7bba590200ff5dae37. Latest consensus state: current_term: 1 leader_uuid: "c20bcd3b33f34c7bba590200ff5dae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20bcd3b33f34c7bba590200ff5dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } } }
I20260621 02:38:46.949173 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:46.956825 5817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:46.964814 5817 catalog_manager.cc:1347] Generated new cluster ID: 8fc44b09fddc4116b735f896cdc58dd7
I20260621 02:38:46.964921 5817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:47.006798 5817 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:47.008569 5817 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:47.018693 5817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c20bcd3b33f34c7bba590200ff5dae37: Generated new TSK 0
I20260621 02:38:47.019639 5817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:38:47.042104 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:47.033705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48406 (local address 127.0.97.62:40941)
0621 02:38:47.033876 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:47.033883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:47.034435 (+ 552us) server_negotiation.cc:407] Connection header received
0621 02:38:47.034631 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:47.034637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:47.034715 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:47.034873 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:38:47.035280 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.036351 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.037765 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.038428 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.038555 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:47.039001 (+ 446us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:38:47.041159 (+ 2158us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33541/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 02:38:47.041767 (+ 608us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33541/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260621 02:38:47.042220 5806 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33541/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260621 02:38:47.044029 5832 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:38:47.033693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40941 (local address 127.0.0.1:48406)
0621 02:38:47.034341 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:47.034360 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:47.034472 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:47.034885 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:47.034888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:47.034897 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:38:47.035141 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:47.035148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.036511 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.036515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:47.037638 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:47.037649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.038551 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.038608 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:47.038728 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:47.038758 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:38:47.041411 (+ 2653us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33541/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 02:38:47.043589 (+ 2178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33541/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":1850,"thread_start_us":376,"threads_started":1}
I20260621 02:38:47.045984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 5737
I20260621 02:38:47.047073 5798 generic_service.cc:196] Checking for leaks (request via RPC)
E20260621 02:38:47.053785 5833 webserver.cc:533] Webserver: sslize failed:
E20260621 02:38:47.053866 5833 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 02:38:47.136155 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 5737
2026-06-21T02:38:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (709 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T02:38:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:47Z Disabled control of system clock
I20260621 02:38:47.214375 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34299
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34299
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/testchainca.cert with env {}
W20260621 02:38:47.526839 5838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:47.527122 5838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:47.527273 5838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:47.534276 5838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:47.534365 5838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:47.534449 5838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:47.534528 5838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:47.534614 5838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:38:47.534724 5838 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:38:47.534785 5838 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 02:38:47.544173 5838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:47.545392 5838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:47.546610 5838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:47.557806 5844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:47.557854 5846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:47.557885 5843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:47.559036 5838 server_base.cc:1034] running on GCE node
I20260621 02:38:47.559829 5838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:47.561160 5838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:47.562366 5838 hybrid_clock.cc:648] HybridClock initialized: now 1782009527562325 us; error 26 us; skew 500 ppm
I20260621 02:38:47.562705 5838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:38:47.564854 5838 webserver.cc:460] Webserver started at http://127.0.97.62:44649/ using document root <none> and password file <none>
I20260621 02:38:47.565552 5838 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:47.565666 5838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:47.565987 5838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:47.568840 5838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "b1fae25e808a42e6b4ec467c2cc32559"
format_stamp: "Formatted at 2026-06-21 02:38:47 on dist-test-slave-jkn7"
server_key: "8775b715306450e9c81d4ebb1818c66a"
server_key_iv: "0b42d400f50074bd2e01ca08d0e90eb2"
server_key_version: "encryptionkey@0"
I20260621 02:38:47.569718 5838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "b1fae25e808a42e6b4ec467c2cc32559"
format_stamp: "Formatted at 2026-06-21 02:38:47 on dist-test-slave-jkn7"
server_key: "8775b715306450e9c81d4ebb1818c66a"
server_key_iv: "0b42d400f50074bd2e01ca08d0e90eb2"
server_key_version: "encryptionkey@0"
I20260621 02:38:47.574430 5838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:38:47.577618 5852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:47.578821 5838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:38:47.579025 5838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "b1fae25e808a42e6b4ec467c2cc32559"
format_stamp: "Formatted at 2026-06-21 02:38:47 on dist-test-slave-jkn7"
server_key: "8775b715306450e9c81d4ebb1818c66a"
server_key_iv: "0b42d400f50074bd2e01ca08d0e90eb2"
server_key_version: "encryptionkey@0"
I20260621 02:38:47.579237 5838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:47.624511 5838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:47.625442 5838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:47.625720 5838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:47.643498 5838 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34299
I20260621 02:38:47.643534 5903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34299 every 8 connection(s)
I20260621 02:38:47.644753 5838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:47.648274 5904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:47.651194 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 5838
I20260621 02:38:47.651396 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:47.652042 388 external_mini_cluster.cc:1428] Setting key ad5f9d3f1a4e7ac3e23764913232ec40
I20260621 02:38:47.660965 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559: Bootstrap starting.
I20260621 02:38:47.664700 5904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:47.665940 5904 log.cc:824] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:47.666386 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:47.653861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.97.62:34299)
0621 02:38:47.654874 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:47.654888 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:47.654946 (+ 58us) server_negotiation.cc:407] Connection header received
0621 02:38:47.656743 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:47.656773 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:47.657386 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:47.657799 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:47.658302 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.660204 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.661524 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.662510 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.662707 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:47.663781 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:47.663791 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:47.663814 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:38:47.664454 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:47.664547 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:38:47.664580 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:47.664702 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:47.664848 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 02:38:47.665187 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":310,"threads_started":1}
I20260621 02:38:47.668810 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559: No bootstrap required, opened a new log
I20260621 02:38:47.669945 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:38:47.675942 5904 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1fae25e808a42e6b4ec467c2cc32559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34299 } }
I20260621 02:38:47.676275 5904 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:47.676383 5904 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1fae25e808a42e6b4ec467c2cc32559, State: Initialized, Role: FOLLOWER
I20260621 02:38:47.677026 5904 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1fae25e808a42e6b4ec467c2cc32559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34299 } }
I20260621 02:38:47.677241 5904 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:47.677388 5904 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:47.677520 5904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:47.679425 5904 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1fae25e808a42e6b4ec467c2cc32559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34299 } }
I20260621 02:38:47.679822 5904 leader_election.cc:302] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1fae25e808a42e6b4ec467c2cc32559; no voters:
I20260621 02:38:47.680335 5904 leader_election.cc:288] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:47.680473 5909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:47.681130 5909 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [term 1 LEADER]: Becoming Leader. State: Replica: b1fae25e808a42e6b4ec467c2cc32559, State: Running, Role: LEADER
I20260621 02:38:47.681764 5909 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1fae25e808a42e6b4ec467c2cc32559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34299 } }
I20260621 02:38:47.682533 5904 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:47.684578 5911 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1fae25e808a42e6b4ec467c2cc32559. Latest consensus state: current_term: 1 leader_uuid: "b1fae25e808a42e6b4ec467c2cc32559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1fae25e808a42e6b4ec467c2cc32559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34299 } } }
I20260621 02:38:47.684561 5910 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1fae25e808a42e6b4ec467c2cc32559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1fae25e808a42e6b4ec467c2cc32559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34299 } } }
I20260621 02:38:47.684958 5911 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:47.684958 5910 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:47.693137 5915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:47.696995 5915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:47.704545 5915 catalog_manager.cc:1347] Generated new cluster ID: e8eea4384f6d4d03a0c8f4c5edb09851
I20260621 02:38:47.704650 5915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:47.722261 5915 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:47.723429 5915 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:47.732455 5915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1fae25e808a42e6b4ec467c2cc32559: Generated new TSK 0
I20260621 02:38:47.733417 5915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:38:47.794350 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:47.751293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.0.97.62:34299)
0621 02:38:47.751486 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:47.751493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:47.753855 (+ 2362us) server_negotiation.cc:407] Connection header received
0621 02:38:47.754031 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:47.754036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:47.754122 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:47.754288 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:38:47.754723 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.755838 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.757203 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.757937 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.758061 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:47.758493 (+ 432us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:38:47.793633 (+ 35140us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 02:38:47.794016 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54250: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260621 02:38:47.794416 5933 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:38:47.751167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34299 (local address 127.0.0.1:54250)
0621 02:38:47.753747 (+ 2580us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:47.753763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:47.753885 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:47.754332 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:47.754336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:47.754346 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:38:47.754580 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:47.754588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.756002 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.756006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:47.757061 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:47.757072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:47.758050 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:47.758095 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:47.758214 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:47.758244 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:38:47.793906 (+ 35662us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 02:38:47.794135 (+ 229us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34299: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":2446,"thread_start_us":2293,"threads_started":1}
W20260621 02:38:47.794490 5907 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54250: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260621 02:38:47.795976 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 5838
I20260621 02:38:47.797068 5899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:47.890642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 5838
2026-06-21T02:38:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (734 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (395 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (394 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:48 dist-test-slave-jkn7 krb5kdc[5948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:48 dist-test-slave-jkn7 krb5kdc[5948](info): set up 2 sockets
Jun 21 02:38:48 dist-test-slave-jkn7 krb5kdc[5948](info): commencing operation
krb5kdc: starting...
W20260621 02:38:49.769881 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:49 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:38:49.964813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42603
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:50.280335 5961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:50.280627 5961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:50.280714 5961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:50.287737 5961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:50.287835 5961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:50.287883 5961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:50.287930 5961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:50.287973 5961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:50.297148 5961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:50.298357 5961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:50.299576 5961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:50.310743 5966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:50.310825 5969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:50.310868 5967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:50.311702 5961 server_base.cc:1034] running on GCE node
I20260621 02:38:50.312474 5961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:50.313838 5961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:50.315058 5961 hybrid_clock.cc:648] HybridClock initialized: now 1782009530315023 us; error 57 us; skew 500 ppm
I20260621 02:38:50.315394 5961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:50 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:50.318454 5961 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:38:50.319957 5961 webserver.cc:460] Webserver started at http://127.0.97.62:41175/ using document root <none> and password file <none>
I20260621 02:38:50.320621 5961 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:50.320763 5961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:50.321076 5961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:50.323977 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "2551b524a5e14e57affc2c790ae106a2"
format_stamp: "Formatted at 2026-06-21 02:38:50 on dist-test-slave-jkn7"
server_key: "dabb378d975d5e32be9d9da384f0ebd0"
server_key_iv: "2cb6cfbfc13657701a819f2cc84cc97a"
server_key_version: "encryptionkey@0"
I20260621 02:38:50.324743 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "2551b524a5e14e57affc2c790ae106a2"
format_stamp: "Formatted at 2026-06-21 02:38:50 on dist-test-slave-jkn7"
server_key: "dabb378d975d5e32be9d9da384f0ebd0"
server_key_iv: "2cb6cfbfc13657701a819f2cc84cc97a"
server_key_version: "encryptionkey@0"
I20260621 02:38:50.329712 5961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:38:50.332983 5976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:50.334302 5961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:50.334493 5961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "2551b524a5e14e57affc2c790ae106a2"
format_stamp: "Formatted at 2026-06-21 02:38:50 on dist-test-slave-jkn7"
server_key: "dabb378d975d5e32be9d9da384f0ebd0"
server_key_iv: "2cb6cfbfc13657701a819f2cc84cc97a"
server_key_version: "encryptionkey@0"
I20260621 02:38:50.334686 5961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:50.349802 5961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:50.353816 5961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:50.354120 5961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:50.372062 5961 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42603
I20260621 02:38:50.372141 6027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42603 every 8 connection(s)
I20260621 02:38:50.373450 5961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:50.373975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 5961
I20260621 02:38:50.374181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:50.375008 388 external_mini_cluster.cc:1428] Setting key f0911da7bd77741894b7b789aedac1fa
I20260621 02:38:50.377710 6028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:50.387380 6028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2: Bootstrap starting.
Jun 21 02:38:50 dist-test-slave-jkn7 krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:50.390239 6028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:50.391222 6028 log.cc:824] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:50.394196 6028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2: No bootstrap required, opened a new log
I20260621 02:38:50.395731 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:50.376859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.0.97.62:42603)
0621 02:38:50.377590 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:50.377601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:50.377638 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:38:50.379220 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:50.379240 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:50.379578 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:50.379964 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:50.381264 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:50.382630 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:50.383823 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:50.384461 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:50.384645 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:50.388354 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:50.388361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:50.388375 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:50.388411 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:50.391453 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:50.392232 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:50.392238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:50.392243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:50.392441 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:50.392808 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:50.392812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:50.392815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:50.393480 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:50.393741 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:50.394063 (+ 322us) server_negotiation.cc:299] Negotiation successful
0621 02:38:50.394455 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":328,"threads_started":1}
I20260621 02:38:50.402223 6028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2551b524a5e14e57affc2c790ae106a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42603 } }
I20260621 02:38:50.402519 6028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:50.402588 6028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2551b524a5e14e57affc2c790ae106a2, State: Initialized, Role: FOLLOWER
I20260621 02:38:50.403053 6028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2551b524a5e14e57affc2c790ae106a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42603 } }
I20260621 02:38:50.403241 6028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:50.403323 6028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:50.403434 6028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:50.405602 6028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2551b524a5e14e57affc2c790ae106a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42603 } }
I20260621 02:38:50.406020 6028 leader_election.cc:302] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2551b524a5e14e57affc2c790ae106a2; no voters:
I20260621 02:38:50.406508 6028 leader_election.cc:288] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:50.406651 6033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:50.407307 6033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [term 1 LEADER]: Becoming Leader. State: Replica: 2551b524a5e14e57affc2c790ae106a2, State: Running, Role: LEADER
I20260621 02:38:50.407750 6033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2551b524a5e14e57affc2c790ae106a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42603 } }
I20260621 02:38:50.408391 6028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:50.410713 6034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2551b524a5e14e57affc2c790ae106a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2551b524a5e14e57affc2c790ae106a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42603 } } }
I20260621 02:38:50.411137 6034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:50.411633 6035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2551b524a5e14e57affc2c790ae106a2. Latest consensus state: current_term: 1 leader_uuid: "2551b524a5e14e57affc2c790ae106a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2551b524a5e14e57affc2c790ae106a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42603 } } }
I20260621 02:38:50.411890 6035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:50.413250 6042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:50.418097 6042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:50.426229 6042 catalog_manager.cc:1347] Generated new cluster ID: c4f19330cff849b2b2f7d5302f3ca7bb
I20260621 02:38:50.426373 6042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:50.438489 6042 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:50.440181 6042 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:50.446934 6042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2551b524a5e14e57affc2c790ae106a2: Generated new TSK 0
I20260621 02:38:50.447633 6042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:38:50.498335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42603
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:50.839032 6054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:50.839358 6054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:50.839461 6054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:50.846562 6054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:50.846655 6054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:50.846758 6054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:50.855849 6054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:50.857126 6054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:50.858418 6054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:50.870698 6059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:50.870663 6060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:50.870744 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:50.871829 6054 server_base.cc:1034] running on GCE node
I20260621 02:38:50.872403 6054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:50.873224 6054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:50.874466 6054 hybrid_clock.cc:648] HybridClock initialized: now 1782009530874420 us; error 35 us; skew 500 ppm
I20260621 02:38:50.874778 6054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:50 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:50.877907 6054 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:38:50.879333 6054 webserver.cc:460] Webserver started at http://127.0.97.1:39759/ using document root <none> and password file <none>
I20260621 02:38:50.879981 6054 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:50.880123 6054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:50.880412 6054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:50.883229 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "8ccada377e9c4cb1ba87c4170c7ff2e3"
format_stamp: "Formatted at 2026-06-21 02:38:50 on dist-test-slave-jkn7"
server_key: "6248d4d76bfd898e8a22973660031db9"
server_key_iv: "52ab9f4719926743438b2d4fac468b46"
server_key_version: "encryptionkey@0"
I20260621 02:38:50.883991 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8ccada377e9c4cb1ba87c4170c7ff2e3"
format_stamp: "Formatted at 2026-06-21 02:38:50 on dist-test-slave-jkn7"
server_key: "6248d4d76bfd898e8a22973660031db9"
server_key_iv: "52ab9f4719926743438b2d4fac468b46"
server_key_version: "encryptionkey@0"
I20260621 02:38:50.888885 6054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:38:50.892119 6069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:50.893420 6054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:50.893581 6054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "8ccada377e9c4cb1ba87c4170c7ff2e3"
format_stamp: "Formatted at 2026-06-21 02:38:50 on dist-test-slave-jkn7"
server_key: "6248d4d76bfd898e8a22973660031db9"
server_key_iv: "52ab9f4719926743438b2d4fac468b46"
server_key_version: "encryptionkey@0"
I20260621 02:38:50.893770 6054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:50.909969 6054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:50.914389 6054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:50.914657 6054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:50.915582 6054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:50.916836 6054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:50.916906 6054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:50.917065 6054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:50.917131 6054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:50.947818 6054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34679
I20260621 02:38:50.947886 6181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34679 every 8 connection(s)
I20260621 02:38:50.949116 6054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:50.958451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 6054
I20260621 02:38:50.958647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:50.959309 388 external_mini_cluster.cc:1428] Setting key 4862fefd41d7a3a4a008bd1c4a293793
Jun 21 02:38:50 dist-test-slave-jkn7 krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:50.968434 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:50.951645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43069 (local address 127.0.97.62:42603)
0621 02:38:50.952059 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:50.952065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:50.953082 (+ 1017us) server_negotiation.cc:407] Connection header received
0621 02:38:50.954821 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:50.954825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:50.954887 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:50.954981 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:50.956301 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:50.956984 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:50.958929 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:50.959304 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:50.959446 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:50.963063 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:50.963069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:50.963072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:50.963101 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:50.965601 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:50.966281 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:50.966292 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:50.966295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:50.966403 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:50.966821 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:50.966825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:50.966828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:50.967077 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:50.967223 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:50.967866 (+ 643us) server_negotiation.cc:299] Negotiation successful
0621 02:38:50.968070 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":191,"threads_started":1}
I20260621 02:38:50.969389 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:50.952215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42603 (local address 127.0.97.1:43069)
0621 02:38:50.952925 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:50.952964 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:50.954589 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:50.955168 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:50.955176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:50.955639 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:50.956157 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:50.956172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:50.957134 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:50.957138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:50.958791 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:50.958807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:50.959537 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:50.959545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:50.959824 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:50.960548 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:50.960569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:50.962843 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:50.965736 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:50.965745 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:50.965761 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:50.966156 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:50.966518 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:50.966524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:50.966527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:50.966705 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:50.967197 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:50.967204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:50.967515 (+ 311us) client_negotiation.cc:769] Sending connection context
0621 02:38:50.967770 (+ 255us) client_negotiation.cc:240] Negotiation successful
0621 02:38:50.968265 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":299,"threads_started":1}
I20260621 02:38:50.970840 6182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42603
I20260621 02:38:50.971128 6182 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:50.971809 6182 heartbeater.cc:511] Master 127.0.97.62:42603 requested a full tablet report, sending...
I20260621 02:38:50.974016 5993 ts_manager.cc:194] Registered new tserver with Master: 8ccada377e9c4cb1ba87c4170c7ff2e3 (127.0.97.1:34679)
I20260621 02:38:50.976238 5993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43069
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:38:51.002437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42603
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:51.316825 6188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:51.317176 6188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:51.317402 6188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:51.324766 6188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:51.324931 6188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:51.325124 6188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:51.335109 6188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:51.336488 6188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:51.338013 6188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:51.351438 6196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:51.351514 6193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:51.351514 6194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:51.352631 6188 server_base.cc:1034] running on GCE node
I20260621 02:38:51.353283 6188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:51.354316 6188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:51.355507 6188 hybrid_clock.cc:648] HybridClock initialized: now 1782009531355445 us; error 50 us; skew 500 ppm
I20260621 02:38:51.355911 6188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:51 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:51.359853 6188 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:38:51.361589 6188 webserver.cc:460] Webserver started at http://127.0.97.2:36879/ using document root <none> and password file <none>
I20260621 02:38:51.362210 6188 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:51.362304 6188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:51.362548 6188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:51.365270 6188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "d113d34cef584902ab05b911a5ac86b4"
format_stamp: "Formatted at 2026-06-21 02:38:51 on dist-test-slave-jkn7"
server_key: "fc2fd509f24ef913442e42e0fbee8393"
server_key_iv: "8aa403ccf3c374d0a6207c26ccd761e7"
server_key_version: "encryptionkey@0"
I20260621 02:38:51.366048 6188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d113d34cef584902ab05b911a5ac86b4"
format_stamp: "Formatted at 2026-06-21 02:38:51 on dist-test-slave-jkn7"
server_key: "fc2fd509f24ef913442e42e0fbee8393"
server_key_iv: "8aa403ccf3c374d0a6207c26ccd761e7"
server_key_version: "encryptionkey@0"
I20260621 02:38:51.370833 6188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260621 02:38:51.374895 6203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:51.376349 6188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:38:51.376502 6188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "d113d34cef584902ab05b911a5ac86b4"
format_stamp: "Formatted at 2026-06-21 02:38:51 on dist-test-slave-jkn7"
server_key: "fc2fd509f24ef913442e42e0fbee8393"
server_key_iv: "8aa403ccf3c374d0a6207c26ccd761e7"
server_key_version: "encryptionkey@0"
I20260621 02:38:51.376713 6188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:51.402642 6188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:51.406280 6188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:51.406517 6188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:51.407352 6188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:51.408573 6188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:51.408643 6188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:51.408722 6188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:51.408764 6188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:51.440668 6188 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35769
I20260621 02:38:51.440760 6315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35769 every 8 connection(s)
I20260621 02:38:51.441932 6188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:51.450552 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 6188
I20260621 02:38:51.450713 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:51.451247 388 external_mini_cluster.cc:1428] Setting key d605ff23d864d3396e0468cad1c4a9b9
Jun 21 02:38:51 dist-test-slave-jkn7 krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:51.462154 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:51.444538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48699 (local address 127.0.97.62:42603)
0621 02:38:51.444730 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:51.444737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:51.445917 (+ 1180us) server_negotiation.cc:407] Connection header received
0621 02:38:51.447613 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:51.447618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:51.447685 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:51.447775 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:51.449202 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:51.449988 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:51.452169 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:51.452815 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:51.452937 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:51.456290 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:51.456295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:51.456298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:51.456330 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:51.459227 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:51.459975 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:51.459982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:51.459988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:51.460118 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:51.460508 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:51.460516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:51.460519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:51.460762 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:51.460904 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:51.461606 (+ 702us) server_negotiation.cc:299] Negotiation successful
0621 02:38:51.461806 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":66}
I20260621 02:38:51.463042 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:51.445021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42603 (local address 127.0.97.2:48699)
0621 02:38:51.445758 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:51.445796 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:51.447349 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:51.447973 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:51.447980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:51.448445 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:51.448976 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:51.448992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:51.450152 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:51.450163 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:51.452015 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:51.452026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:51.453057 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:51.453066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:51.453347 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:51.454039 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:51.454059 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:51.456085 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:51.459407 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:51.459416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:51.459433 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:51.459855 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:51.460223 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:51.460229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:51.460233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:51.460395 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:51.460882 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:51.460889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:51.461189 (+ 300us) client_negotiation.cc:769] Sending connection context
0621 02:38:51.461525 (+ 336us) client_negotiation.cc:240] Negotiation successful
0621 02:38:51.462015 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":338,"threads_started":1}
I20260621 02:38:51.464386 6316 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42603
I20260621 02:38:51.464677 6316 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:51.465391 6316 heartbeater.cc:511] Master 127.0.97.62:42603 requested a full tablet report, sending...
I20260621 02:38:51.466779 5993 ts_manager.cc:194] Registered new tserver with Master: d113d34cef584902ab05b911a5ac86b4 (127.0.97.2:35769)
I20260621 02:38:51.467515 5993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48699
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:38:51.494438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42603
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:51.857883 6321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:51.858137 6321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:51.858280 6321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:51.865240 6321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:51.865407 6321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:51.865597 6321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:51.875241 6321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:51.876483 6321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:51.877914 6321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:51.894001 6326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:51.894150 6327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:51.894991 6329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:51.895386 6321 server_base.cc:1034] running on GCE node
I20260621 02:38:51.896157 6321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:51.897537 6321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:51.898790 6321 hybrid_clock.cc:648] HybridClock initialized: now 1782009531898691 us; error 104 us; skew 500 ppm
I20260621 02:38:51.899219 6321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:51 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:51.907850 6321 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:38:51.909613 6321 webserver.cc:460] Webserver started at http://127.0.97.3:41223/ using document root <none> and password file <none>
I20260621 02:38:51.910392 6321 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:51.910526 6321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:51.910851 6321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:51.913959 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "38eb5416ca4943309afdc0a9910edfa2"
format_stamp: "Formatted at 2026-06-21 02:38:51 on dist-test-slave-jkn7"
server_key: "f12ceb27edce862429644f019605cbdd"
server_key_iv: "3a9d35829b908cc78795bf8d911c9dbd"
server_key_version: "encryptionkey@0"
I20260621 02:38:51.914758 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "38eb5416ca4943309afdc0a9910edfa2"
format_stamp: "Formatted at 2026-06-21 02:38:51 on dist-test-slave-jkn7"
server_key: "f12ceb27edce862429644f019605cbdd"
server_key_iv: "3a9d35829b908cc78795bf8d911c9dbd"
server_key_version: "encryptionkey@0"
I20260621 02:38:51.920027 6321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:38:51.923702 6336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:51.925129 6321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:51.925290 6321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "38eb5416ca4943309afdc0a9910edfa2"
format_stamp: "Formatted at 2026-06-21 02:38:51 on dist-test-slave-jkn7"
server_key: "f12ceb27edce862429644f019605cbdd"
server_key_iv: "3a9d35829b908cc78795bf8d911c9dbd"
server_key_version: "encryptionkey@0"
I20260621 02:38:51.925519 6321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:51.942378 6321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:51.946642 6321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:51.946913 6321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:51.947891 6321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:51.949177 6321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:51.949278 6321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:51.949440 6321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:51.949514 6321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:51.979076 6182 heartbeater.cc:503] Master 127.0.97.62:42603 was elected leader, sending a full tablet report...
I20260621 02:38:51.982226 6321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38487
I20260621 02:38:51.982263 6448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38487 every 8 connection(s)
I20260621 02:38:51.983413 6321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:51.993721 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 6321
I20260621 02:38:51.993887 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:51.994310 388 external_mini_cluster.cc:1428] Setting key db06c10dc7e4ac0e034e652bbc2fe1f7
Jun 21 02:38:51 dist-test-slave-jkn7 krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:38:52.001619 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:51.985847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49115 (local address 127.0.97.62:42603)
0621 02:38:51.986273 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:51.986278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:51.987217 (+ 939us) server_negotiation.cc:407] Connection header received
0621 02:38:51.988868 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:51.988873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:51.988933 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:51.989031 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:51.990435 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:51.991178 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:51.992599 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:51.992984 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:51.993071 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:51.996507 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:51.996512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:51.996517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:51.996553 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:51.999133 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:51.999757 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:51.999761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:51.999763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:51.999869 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.000212 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.000215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.000217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.000445 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:52.000555 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:52.001110 (+ 555us) server_negotiation.cc:299] Negotiation successful
0621 02:38:52.001251 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260621 02:38:52.002481 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:51.986396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42603 (local address 127.0.97.3:49115)
0621 02:38:51.987069 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:51.987106 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:51.988633 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:51.989230 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:51.989238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:51.989781 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:51.990292 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:51.990308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:51.991304 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:51.991307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:51.992478 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:51.992498 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:51.993153 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:51.993157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:51.993394 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:51.994116 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:51.994141 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:51.996316 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:51.999249 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:51.999255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:51.999266 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:51.999661 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:51.999957 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:51.999959 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:51.999962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:52.000124 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:52.000549 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:52.000554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:52.000820 (+ 266us) client_negotiation.cc:769] Sending connection context
0621 02:38:52.001040 (+ 220us) client_negotiation.cc:240] Negotiation successful
0621 02:38:52.001474 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":267,"threads_started":1}
I20260621 02:38:52.003813 6449 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42603
I20260621 02:38:52.004060 6449 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:52.004684 6449 heartbeater.cc:511] Master 127.0.97.62:42603 requested a full tablet report, sending...
I20260621 02:38:52.006009 5993 ts_manager.cc:194] Registered new tserver with Master: 38eb5416ca4943309afdc0a9910edfa2 (127.0.97.3:38487)
I20260621 02:38:52.006767 5993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49115
I20260621 02:38:52.009436 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/test-admin.keytab.
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:52.034565 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260621 02:38:52.039484 6457 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:38:52.036993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.0.97.1:34679)
0621 02:38:52.037636 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:52.037647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:52.037682 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:38:52.037763 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:52.037770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:52.037989 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:52.038153 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:52.039056 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60676: BlockingRecv error: recv got EOF from 127.0.0.1:60676 (error 108)
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":298,"threads_started":1}
W20260621 02:38:52.039716 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5cc' of type FILE
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:52.041128 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:52.042479 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:52.056898 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:52.043890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.0.97.1:34679)
0621 02:38:52.044072 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:52.044078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:52.044245 (+ 167us) server_negotiation.cc:407] Connection header received
0621 02:38:52.044403 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:52.044407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:52.044467 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:52.044550 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:52.045526 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.046466 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.047651 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.048224 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.048333 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:52.051827 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:52.051838 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:52.051851 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:52.051883 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:52.054558 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.055154 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.055161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.055163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.055277 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.055595 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.055598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.055600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.055906 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:52.056128 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:52.056374 (+ 246us) server_negotiation.cc:299] Negotiation successful
0621 02:38:52.056623 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260621 02:38:52.062561 6457 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:38:52.060733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.0.97.1:34679)
0621 02:38:52.060969 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:52.060974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:52.061198 (+ 224us) server_negotiation.cc:407] Connection header received
0621 02:38:52.061462 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:52.061468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:52.061536 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:52.061633 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:52.062336 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60702: BlockingRecv error: recv got EOF from 127.0.0.1:60702 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260621 02:38:52.062747 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5cc' of type FILE
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:52.064252 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:52.065595 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:52.078835 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:52.066932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.0.97.1:34679)
0621 02:38:52.067103 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:52.067107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:52.067171 (+ 64us) server_negotiation.cc:407] Connection header received
0621 02:38:52.067319 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:52.067327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:52.067414 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:52.067510 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:52.068397 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.069074 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.070309 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.070719 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.070817 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:52.074070 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:52.074075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:52.074078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:52.074106 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:52.076332 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.076958 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.076962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.076965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.077090 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.077553 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.077557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.077559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.077826 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:52.078014 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:52.078332 (+ 318us) server_negotiation.cc:299] Negotiation successful
0621 02:38:52.078577 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 02:38:52.083240 6457 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:38:52.081483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.0.97.1:34679)
0621 02:38:52.081678 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:52.081685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:52.082020 (+ 335us) server_negotiation.cc:407] Connection header received
0621 02:38:52.082153 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:52.082156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:52.082221 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:52.082299 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:52.083032 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60730: BlockingRecv error: recv got EOF from 127.0.0.1:60730 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260621 02:38:52.083320 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009483336832-388-0/krb5kdc/krb5cc' of type FILE
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:52.084786 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:52.086127 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:38:52.099612 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:52.087284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.0.97.1:34679)
0621 02:38:52.087471 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:52.087480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:52.087700 (+ 220us) server_negotiation.cc:407] Connection header received
0621 02:38:52.087863 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:52.087868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:52.087933 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:52.088029 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:52.088923 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.089701 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.090931 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.091343 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.091468 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:52.094782 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:52.094793 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:52.094796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:52.094824 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:52.097211 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.097828 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.097835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.097839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.097987 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.098345 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.098350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.098353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.098616 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:52.098915 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:52.099116 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 02:38:52.099313 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:38:52.101207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 6054
I20260621 02:38:52.112749 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:52.101932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.0.97.1:34679)
0621 02:38:52.102092 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:52.102096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:52.102418 (+ 322us) server_negotiation.cc:407] Connection header received
0621 02:38:52.102586 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:52.102591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:52.102648 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:52.102742 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:52.103617 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.104251 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.105465 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.105878 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.106004 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:52.108003 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:52.108009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:52.108015 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:52.108049 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:52.110390 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.110989 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.111000 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.111003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.111137 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.111483 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.111489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.111492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.111717 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:52.111981 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:52.112267 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 02:38:52.112467 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:38:52.113258 6177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:52.194171 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 6054
I20260621 02:38:52.210881 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 6188
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:38:52.225662 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:52.211779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.0.97.2:35769)
0621 02:38:52.212360 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:52.212367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:52.212392 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:38:52.212449 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:52.212455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:52.212681 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:52.212853 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:52.213896 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.214892 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.216121 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.216655 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.216803 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:52.220650 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:52.220660 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:52.220674 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:52.220712 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:52.223385 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.223959 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.223965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.223968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.224087 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.224408 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.224411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.224414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.224716 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:52.224934 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:52.225148 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 02:38:52.225389 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
I20260621 02:38:52.226900 6311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:52.306874 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 6188
I20260621 02:38:52.322954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 6321
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:38:52.338308 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:52.323945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.0.97.3:38487)
0621 02:38:52.324511 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:52.324519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:52.324546 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:38:52.324603 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:52.324609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:52.324823 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:52.324947 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:52.325984 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.326984 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.328220 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:52.328826 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:52.328919 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:52.332861 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:52.332871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:52.332885 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:52.332915 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:52.335707 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.336299 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.336305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.336307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.336417 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:52.336772 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:52.336775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:52.336777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:52.337072 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:52.337351 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:52.337716 (+ 365us) server_negotiation.cc:299] Negotiation successful
0621 02:38:52.337979 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":256,"threads_started":1}
I20260621 02:38:52.339560 6444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:52.426848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 6321
I20260621 02:38:52.442575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 5961
I20260621 02:38:52.443799 6023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:52.544772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 5961
2026-06-21T02:38:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3871 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[6477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[6477](info): set up 2 sockets
Jun 21 02:38:52 dist-test-slave-jkn7 krb5kdc[6477](info): commencing operation
krb5kdc: starting...
W20260621 02:38:53.633951 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:38:53 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:38:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260621 02:38:53.830410 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46061
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:54.137099 6490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:54.137427 6490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:54.137589 6490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:54.144402 6490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:54.144536 6490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:54.144642 6490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:54.144745 6490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:54.144831 6490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:54.153945 6490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:54.155126 6490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:54.156729 6490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:54.171274 6496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:54.171360 6498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:54.171739 6495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:54.171952 6490 server_base.cc:1034] running on GCE node
I20260621 02:38:54.172780 6490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:54.174625 6490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:54.175843 6490 hybrid_clock.cc:648] HybridClock initialized: now 1782009534175798 us; error 40 us; skew 500 ppm
I20260621 02:38:54.176188 6490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:54 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:54.180115 6490 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260621 02:38:54.181727 6490 webserver.cc:460] Webserver started at http://127.0.97.62:39409/ using document root <none> and password file <none>
I20260621 02:38:54.182753 6490 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:54.182888 6490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:54.183269 6490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:54.188019 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "1522f392977c491790c78c87c51a46b6"
format_stamp: "Formatted at 2026-06-21 02:38:54 on dist-test-slave-jkn7"
server_key: "b66705f1ca6d0036c401ea578f7d1ea9"
server_key_iv: "9f949b6bddf3db4c8a86ee8820770cf6"
server_key_version: "encryptionkey@0"
I20260621 02:38:54.188881 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "1522f392977c491790c78c87c51a46b6"
format_stamp: "Formatted at 2026-06-21 02:38:54 on dist-test-slave-jkn7"
server_key: "b66705f1ca6d0036c401ea578f7d1ea9"
server_key_iv: "9f949b6bddf3db4c8a86ee8820770cf6"
server_key_version: "encryptionkey@0"
I20260621 02:38:54.196238 6490 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260621 02:38:54.199812 6505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:54.201257 6490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:54.201524 6490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "1522f392977c491790c78c87c51a46b6"
format_stamp: "Formatted at 2026-06-21 02:38:54 on dist-test-slave-jkn7"
server_key: "b66705f1ca6d0036c401ea578f7d1ea9"
server_key_iv: "9f949b6bddf3db4c8a86ee8820770cf6"
server_key_version: "encryptionkey@0"
I20260621 02:38:54.201736 6490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:54.232914 6490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:54.236714 6490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:54.236939 6490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:54.255064 6490 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46061
I20260621 02:38:54.255101 6556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46061 every 8 connection(s)
I20260621 02:38:54.256335 6490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:38:54.258356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 6490
I20260621 02:38:54.258564 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:38:54.259217 388 external_mini_cluster.cc:1428] Setting key 9c4d2fdbe0472a1cee2bc07da5573483
I20260621 02:38:54.260272 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:38:54 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:38:54.273895 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: Bootstrap starting.
I20260621 02:38:54.277944 6557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:54.279207 6557 log.cc:824] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:54.282442 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: No bootstrap required, opened a new log
I20260621 02:38:54.282673 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:54.261136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.0.97.62:46061)
0621 02:38:54.262008 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:54.262030 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:54.262080 (+ 50us) server_negotiation.cc:407] Connection header received
0621 02:38:54.263823 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:54.263851 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:54.264374 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:54.264839 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:54.266127 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:54.268105 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:54.269358 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:54.270360 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:54.270551 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:54.274343 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:54.274353 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:54.274369 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:54.274417 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:54.278358 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:54.279016 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:54.279024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:54.279033 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:54.279234 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:54.279678 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:54.279686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:54.279692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:54.280249 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:54.280522 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:54.280921 (+ 399us) server_negotiation.cc:299] Negotiation successful
0621 02:38:54.281358 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":337,"threads_started":1}
I20260621 02:38:54.290522 6557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } }
I20260621 02:38:54.290874 6557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:54.290992 6557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1522f392977c491790c78c87c51a46b6, State: Initialized, Role: FOLLOWER
I20260621 02:38:54.291565 6557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } }
I20260621 02:38:54.291771 6557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:38:54.291865 6557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:38:54.291986 6557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:54.294435 6557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } }
I20260621 02:38:54.294883 6557 leader_election.cc:302] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1522f392977c491790c78c87c51a46b6; no voters:
I20260621 02:38:54.295499 6557 leader_election.cc:288] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:38:54.295657 6562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:54.296612 6562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 1 LEADER]: Becoming Leader. State: Replica: 1522f392977c491790c78c87c51a46b6, State: Running, Role: LEADER
I20260621 02:38:54.297169 6562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } }
I20260621 02:38:54.297866 6557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:54.299822 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1522f392977c491790c78c87c51a46b6. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } }
I20260621 02:38:54.300119 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:54.300981 6568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:38:54.300835 6563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } }
I20260621 02:38:54.302035 6563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:54.306887 6568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:38:54.316920 6568 catalog_manager.cc:1347] Generated new cluster ID: 0c06c1c544434673a79cfe1a0e53415f
I20260621 02:38:54.317070 6568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:38:54.348503 6568 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:38:54.349982 6568 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:38:54.355382 6568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: Generated new TSK 0
I20260621 02:38:54.356199 6568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260621 02:38:54.415005 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46061
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:54.760556 6583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:54.760819 6583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:54.760910 6583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:54.769316 6583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:54.769431 6583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:54.769550 6583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:54.778753 6583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:54.779871 6583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:54.781184 6583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:54.794142 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:54.794171 6589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:54.794204 6591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:54.795189 6583 server_base.cc:1034] running on GCE node
I20260621 02:38:54.795755 6583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:54.796635 6583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:54.797848 6583 hybrid_clock.cc:648] HybridClock initialized: now 1782009534797810 us; error 35 us; skew 500 ppm
I20260621 02:38:54.798143 6583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:54 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:54.801124 6583 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260621 02:38:54.802657 6583 webserver.cc:460] Webserver started at http://127.0.97.1:32857/ using document root <none> and password file <none>
I20260621 02:38:54.803246 6583 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:54.803340 6583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:54.803623 6583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:54.806484 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "406a3d74c2044d018ddd85d9bbff38ca"
format_stamp: "Formatted at 2026-06-21 02:38:54 on dist-test-slave-jkn7"
server_key: "eefd6b0736f7d3bc28e18d1891f73ea1"
server_key_iv: "322c1c9c746a491c1c796b2f9b5e3c33"
server_key_version: "encryptionkey@0"
I20260621 02:38:54.807235 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "406a3d74c2044d018ddd85d9bbff38ca"
format_stamp: "Formatted at 2026-06-21 02:38:54 on dist-test-slave-jkn7"
server_key: "eefd6b0736f7d3bc28e18d1891f73ea1"
server_key_iv: "322c1c9c746a491c1c796b2f9b5e3c33"
server_key_version: "encryptionkey@0"
I20260621 02:38:54.811923 6583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:38:54.815114 6598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:54.816391 6583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:38:54.816617 6583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "406a3d74c2044d018ddd85d9bbff38ca"
format_stamp: "Formatted at 2026-06-21 02:38:54 on dist-test-slave-jkn7"
server_key: "eefd6b0736f7d3bc28e18d1891f73ea1"
server_key_iv: "322c1c9c746a491c1c796b2f9b5e3c33"
server_key_version: "encryptionkey@0"
I20260621 02:38:54.816824 6583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:54.832700 6583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:54.836488 6583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:54.836764 6583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:54.837777 6583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:54.839022 6583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:54.839093 6583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:54.839237 6583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:54.839318 6583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:54.872972 6583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38313
I20260621 02:38:54.873057 6710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38313 every 8 connection(s)
I20260621 02:38:54.874297 6583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:54.876713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 6583
I20260621 02:38:54.876930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:38:54.877703 388 external_mini_cluster.cc:1428] Setting key c4d7412d1cddf99602cba732bbdd148b
Jun 21 02:38:54 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:38:54.898041 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:54.876849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34987 (local address 127.0.97.62:46061)
0621 02:38:54.877360 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:54.877367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:54.878564 (+ 1197us) server_negotiation.cc:407] Connection header received
0621 02:38:54.880852 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:54.880856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:54.880916 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:54.881004 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:54.883114 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:54.883887 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:54.886427 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:54.886964 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:54.887108 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:54.891129 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:54.891137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:54.891145 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:54.891189 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:54.894687 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:54.895385 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:54.895393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:54.895396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:54.895545 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:54.895989 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:54.895997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:54.896003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:54.896364 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:54.896531 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:54.897228 (+ 697us) server_negotiation.cc:299] Negotiation successful
0621 02:38:54.897506 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260621 02:38:54.898697 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:54.877534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.1:34987)
0621 02:38:54.878404 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:54.878443 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:54.880549 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:54.881313 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:54.881328 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:54.882126 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:54.882941 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:54.882966 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:54.884047 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:54.884055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:54.886232 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:54.886249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:54.887190 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:54.887199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:54.887429 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:54.888114 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:54.888148 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:54.890905 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:54.894831 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:54.894837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:54.894850 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:54.895248 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:54.895649 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:54.895652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:54.895654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:54.895847 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:54.896503 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:54.896509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:54.896821 (+ 312us) client_negotiation.cc:769] Sending connection context
0621 02:38:54.897047 (+ 226us) client_negotiation.cc:240] Negotiation successful
0621 02:38:54.897489 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":370,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260621 02:38:54.900247 6711 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46061
I20260621 02:38:54.900581 6711 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:54.901237 6711 heartbeater.cc:511] Master 127.0.97.62:46061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260621 02:38:54.903640 6522 ts_manager.cc:194] Registered new tserver with Master: 406a3d74c2044d018ddd85d9bbff38ca (127.0.97.1:38313)
I20260621 02:38:54.905802 6522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260621 02:38:54.925928 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46061
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:55.239796 6717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:55.240137 6717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:55.240306 6717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:55.247414 6717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:55.247582 6717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:55.247772 6717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:55.257503 6717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:55.258795 6717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:55.260094 6717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:55.273033 6722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:55.273089 6725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:55.273042 6723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:55.273964 6717 server_base.cc:1034] running on GCE node
I20260621 02:38:55.274607 6717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:55.275589 6717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:55.276808 6717 hybrid_clock.cc:648] HybridClock initialized: now 1782009535276749 us; error 58 us; skew 500 ppm
I20260621 02:38:55.277117 6717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:55 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009535, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:55.280129 6717 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260621 02:38:55.281617 6717 webserver.cc:460] Webserver started at http://127.0.97.2:39673/ using document root <none> and password file <none>
I20260621 02:38:55.282303 6717 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:55.282445 6717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:55.282739 6717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:55.285837 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "7e54a539be224025a030c0272b000fac"
format_stamp: "Formatted at 2026-06-21 02:38:55 on dist-test-slave-jkn7"
server_key: "f6e8621464c9d963ea6123b2b6167dab"
server_key_iv: "e4772b3e6c0eaa6ba09a0db866a8915c"
server_key_version: "encryptionkey@0"
I20260621 02:38:55.286718 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7e54a539be224025a030c0272b000fac"
format_stamp: "Formatted at 2026-06-21 02:38:55 on dist-test-slave-jkn7"
server_key: "f6e8621464c9d963ea6123b2b6167dab"
server_key_iv: "e4772b3e6c0eaa6ba09a0db866a8915c"
server_key_version: "encryptionkey@0"
I20260621 02:38:55.291988 6717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:38:55.295343 6732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:55.296821 6717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:38:55.297025 6717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "7e54a539be224025a030c0272b000fac"
format_stamp: "Formatted at 2026-06-21 02:38:55 on dist-test-slave-jkn7"
server_key: "f6e8621464c9d963ea6123b2b6167dab"
server_key_iv: "e4772b3e6c0eaa6ba09a0db866a8915c"
server_key_version: "encryptionkey@0"
I20260621 02:38:55.297230 6717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:55.332202 6717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:55.336153 6717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:55.336448 6717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:55.337445 6717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:55.338723 6717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:55.338828 6717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:55.338989 6717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:55.339062 6717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:55.374927 6717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45449
I20260621 02:38:55.374981 6844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45449 every 8 connection(s)
I20260621 02:38:55.376262 6717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:55.383603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 6717
I20260621 02:38:55.383780 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:38:55.384214 388 external_mini_cluster.cc:1428] Setting key dcc2483e4ee3f349c04b09989c3c5781
Jun 21 02:38:55 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009535, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:38:55.397186 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:55.378719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52929 (local address 127.0.97.62:46061)
0621 02:38:55.378894 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:55.378898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:55.380045 (+ 1147us) server_negotiation.cc:407] Connection header received
0621 02:38:55.381816 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:55.381821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:55.381886 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:55.381976 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:55.383504 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:55.384356 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:55.386197 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:55.387014 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:55.387171 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:55.390953 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:55.390960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:55.390972 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:55.391004 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:55.393740 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:55.394754 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:55.394758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:55.394760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:55.394905 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:55.395425 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:55.395429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:55.395431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:55.395690 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:55.395806 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:55.396743 (+ 937us) server_negotiation.cc:299] Negotiation successful
0621 02:38:55.396917 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:38:55.398855 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:55.379191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.2:52929)
0621 02:38:55.379876 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:55.379913 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:55.381560 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:55.382182 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:55.382190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:55.382675 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:55.383319 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:55.383349 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:55.384503 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:55.384509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:55.386032 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:55.386044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:55.387274 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:55.387282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:55.387513 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:55.388234 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:55.388261 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:55.390702 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:55.393890 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:55.393902 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:55.393925 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:55.394575 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:55.395035 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:55.395043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:55.395050 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:55.395306 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:55.395816 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:55.395827 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:55.396311 (+ 484us) client_negotiation.cc:769] Sending connection context
0621 02:38:55.396648 (+ 337us) client_negotiation.cc:240] Negotiation successful
0621 02:38:55.397246 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":276,"threads_started":1}
I20260621 02:38:55.400347 6845 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46061
I20260621 02:38:55.400691 6845 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:55.401685 6845 heartbeater.cc:511] Master 127.0.97.62:46061 requested a full tablet report, sending...
I20260621 02:38:55.403358 6522 ts_manager.cc:194] Registered new tserver with Master: 7e54a539be224025a030c0272b000fac (127.0.97.2:45449)
I20260621 02:38:55.404225 6522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52929
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260621 02:38:55.431988 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46061
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:55.745397 6850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:55.745694 6850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:55.745864 6850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:55.752719 6850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:55.752856 6850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:55.753037 6850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:55.762267 6850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:55.763489 6850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:55.764767 6850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:55.778617 6855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:55.778755 6856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:55.778770 6858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:55.779518 6850 server_base.cc:1034] running on GCE node
I20260621 02:38:55.780103 6850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:55.782109 6850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:55.783372 6850 hybrid_clock.cc:648] HybridClock initialized: now 1782009535783297 us; error 59 us; skew 500 ppm
I20260621 02:38:55.783717 6850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:55 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009535, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:55.786868 6850 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260621 02:38:55.788328 6850 webserver.cc:460] Webserver started at http://127.0.97.3:38715/ using document root <none> and password file <none>
I20260621 02:38:55.788990 6850 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:55.789141 6850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:55.789502 6850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:55.792640 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "26fe039cc5604f76850fe38274b4ef3f"
format_stamp: "Formatted at 2026-06-21 02:38:55 on dist-test-slave-jkn7"
server_key: "c7ef9c2adfd99887c35e0a4fd821490b"
server_key_iv: "6f5f72801956d66a04821d174eb659b5"
server_key_version: "encryptionkey@0"
I20260621 02:38:55.793498 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "26fe039cc5604f76850fe38274b4ef3f"
format_stamp: "Formatted at 2026-06-21 02:38:55 on dist-test-slave-jkn7"
server_key: "c7ef9c2adfd99887c35e0a4fd821490b"
server_key_iv: "6f5f72801956d66a04821d174eb659b5"
server_key_version: "encryptionkey@0"
I20260621 02:38:55.798651 6850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:38:55.802575 6865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:55.804104 6850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:38:55.804296 6850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "26fe039cc5604f76850fe38274b4ef3f"
format_stamp: "Formatted at 2026-06-21 02:38:55 on dist-test-slave-jkn7"
server_key: "c7ef9c2adfd99887c35e0a4fd821490b"
server_key_iv: "6f5f72801956d66a04821d174eb659b5"
server_key_version: "encryptionkey@0"
I20260621 02:38:55.804534 6850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:55.822618 6850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:55.826985 6850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:55.827301 6850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:55.828263 6850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:55.829674 6850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:55.829777 6850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:55.829928 6850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:55.830005 6850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:55.862910 6850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43879
I20260621 02:38:55.862957 6977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43879 every 8 connection(s)
I20260621 02:38:55.864176 6850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:55.871632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 6850
I20260621 02:38:55.871794 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:38:55.872258 388 external_mini_cluster.cc:1428] Setting key edc5b600f5f3b2ade9742065f20b6321
Jun 21 02:38:55 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009535, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:38:55.884550 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:55.867073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57227 (local address 127.0.97.62:46061)
0621 02:38:55.867254 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:55.867259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:55.868541 (+ 1282us) server_negotiation.cc:407] Connection header received
0621 02:38:55.870367 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:55.870373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:55.870455 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:55.870577 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:55.872063 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:55.873199 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:55.875402 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:55.875803 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:55.875934 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:55.879283 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:55.879296 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:55.879300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:55.879330 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:55.881699 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:55.882413 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:55.882420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:55.882424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:55.882545 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:55.882961 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:55.882966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:55.882969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:55.883217 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:55.883361 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:55.883988 (+ 627us) server_negotiation.cc:299] Negotiation successful
0621 02:38:55.884196 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:38:55.885483 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:55.867538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.3:57227)
0621 02:38:55.868377 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:55.868424 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:55.870109 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:55.870800 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:55.870809 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:55.871272 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:55.871905 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:55.871925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:55.873520 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:55.873526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:55.875252 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:55.875266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:55.876030 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:55.876037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:55.876269 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:55.876946 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:55.876968 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:55.879083 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:55.881830 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:55.881839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:55.881853 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:55.882281 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:55.882653 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:55.882658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:55.882662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:55.882836 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:55.883348 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:55.883356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:55.883648 (+ 292us) client_negotiation.cc:769] Sending connection context
0621 02:38:55.883902 (+ 254us) client_negotiation.cc:240] Negotiation successful
0621 02:38:55.884379 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":361,"threads_started":1}
I20260621 02:38:55.886942 6978 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46061
I20260621 02:38:55.887240 6978 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:55.887929 6978 heartbeater.cc:511] Master 127.0.97.62:46061 requested a full tablet report, sending...
I20260621 02:38:55.889232 6521 ts_manager.cc:194] Registered new tserver with Master: 26fe039cc5604f76850fe38274b4ef3f (127.0.97.3:43879)
I20260621 02:38:55.890038 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57227
I20260621 02:38:55.899571 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:38:55.908591 6711 heartbeater.cc:503] Master 127.0.97.62:46061 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260621 02:38:55.943022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40413
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46061,127.0.97.61:40413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:38:56.255210 6983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:56.255538 6983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:56.255707 6983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:56.263446 6983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:56.263588 6983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:56.263691 6983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:56.263792 6983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:56.263864 6983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:56.273552 6983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46061,127.0.97.61:40413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
W20260621 02:38:56.273922 6983 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:46061,127.0.97.61:40413'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 02:38:56.274935 6983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:56.276222 6983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:56.287349 6989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:56.287379 6991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:56.287438 6988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:56.288745 6983 server_base.cc:1034] running on GCE node
I20260621 02:38:56.289474 6983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:56.290789 6983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:56.292004 6983 hybrid_clock.cc:648] HybridClock initialized: now 1782009536291948 us; error 31 us; skew 500 ppm
I20260621 02:38:56.292315 6983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:56 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009536, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:56.295445 6983 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260621 02:38:56.296823 6983 webserver.cc:460] Webserver started at http://127.0.97.61:43511/ using document root <none> and password file <none>
I20260621 02:38:56.297626 6983 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:56.297745 6983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:56.298069 6983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:56.300933 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data/instance:
uuid: "d8b7d45d78264253924c1d843c95af9a"
format_stamp: "Formatted at 2026-06-21 02:38:56 on dist-test-slave-jkn7"
server_key: "8a63e6c24c3f81ab92f256921e25bf32"
server_key_iv: "3e70cb55358b898d5d5892c113cb4bb2"
server_key_version: "encryptionkey@0"
I20260621 02:38:56.301803 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal/instance:
uuid: "d8b7d45d78264253924c1d843c95af9a"
format_stamp: "Formatted at 2026-06-21 02:38:56 on dist-test-slave-jkn7"
server_key: "8a63e6c24c3f81ab92f256921e25bf32"
server_key_iv: "3e70cb55358b898d5d5892c113cb4bb2"
server_key_version: "encryptionkey@0"
I20260621 02:38:56.307106 6983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:38:56.310763 6998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:56.312167 6983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:38:56.312352 6983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal
uuid: "d8b7d45d78264253924c1d843c95af9a"
format_stamp: "Formatted at 2026-06-21 02:38:56 on dist-test-slave-jkn7"
server_key: "8a63e6c24c3f81ab92f256921e25bf32"
server_key_iv: "3e70cb55358b898d5d5892c113cb4bb2"
server_key_version: "encryptionkey@0"
I20260621 02:38:56.312566 6983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:56.334722 6983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:56.339182 6983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:56.339464 6983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:56.357833 6983 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40413
I20260621 02:38:56.357853 7049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40413 every 8 connection(s)
I20260621 02:38:56.359230 6983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
I20260621 02:38:56.360101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 6983
I20260621 02:38:56.364614 7050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:56.374166 7050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:38:56 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 21 02:38:56 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009536, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:38:56.391461 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.366256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.0.97.61:40413)
0621 02:38:56.369840 (+ 3584us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:56.369855 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:56.369907 (+ 52us) server_negotiation.cc:407] Connection header received
0621 02:38:56.371545 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:56.371572 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:56.372085 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:56.372516 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:56.374193 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.376067 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.377890 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.378871 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.379086 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.383783 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:56.383790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:56.383801 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:56.383839 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:56.387146 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.387881 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.387891 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.387901 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.388085 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.388448 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.388453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.388457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.389163 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:56.389489 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:56.389877 (+ 388us) server_negotiation.cc:299] Negotiation successful
0621 02:38:56.390289 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3331,"thread_start_us":3097,"threads_started":1}
I20260621 02:38:56.393052 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.377087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.0.97.62:46061)
0621 02:38:56.377336 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:56.377344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:56.378030 (+ 686us) server_negotiation.cc:407] Connection header received
0621 02:38:56.378362 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:56.378367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:56.378433 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:56.378545 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:56.379900 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.381096 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.382935 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.383350 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.383501 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.386976 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:56.386982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:56.386985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:56.387017 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:56.390301 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.391137 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.391142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.391146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.391278 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.391674 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.391680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.391683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.392071 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:56.392221 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:56.392655 (+ 434us) server_negotiation.cc:299] Negotiation successful
0621 02:38:56.392840 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:38:56.393160 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.376983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60156)
0621 02:38:56.377889 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:56.377928 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:56.378228 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:56.378592 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:56.378598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:56.379418 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:56.379694 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.379702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.381320 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.381328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.382784 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.382795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.383585 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.383592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.383744 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.384384 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:56.384400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:56.386770 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:56.390435 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.390443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.390448 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.391006 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.391378 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.391381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.391383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.391507 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.392218 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:56.392223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:56.392352 (+ 129us) client_negotiation.cc:769] Sending connection context
0621 02:38:56.392504 (+ 152us) client_negotiation.cc:240] Negotiation successful
0621 02:38:56.392731 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":551,"threads_started":1}
I20260621 02:38:56.395102 7050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:38:56 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009536, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260621 02:38:56.407303 6845 heartbeater.cc:503] Master 127.0.97.62:46061 was elected leader, sending a full tablet report...
I20260621 02:38:56.409312 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.395829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.0.1:36894)
0621 02:38:56.396019 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:56.396038 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:56.396127 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:56.396370 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:56.396373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:56.396674 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:56.396793 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.396800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.397950 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.397955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.398825 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.398835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.399690 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.399694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.399851 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.400510 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:56.400527 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:56.402993 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:56.406774 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.406778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.406781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.407198 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.407561 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.407565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.407567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.407717 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.408345 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:56.408349 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:56.408514 (+ 165us) client_negotiation.cc:769] Sending connection context
0621 02:38:56.408672 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 02:38:56.408963 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260621 02:38:56.409587 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.395893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.0.97.61:40413)
0621 02:38:56.396072 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:56.396076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:56.396090 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:38:56.396214 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:56.396217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:56.396265 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:56.396338 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:56.396950 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.397722 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.399015 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.399544 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.399695 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.403189 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:56.403197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:56.403205 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:56.403246 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:56.406643 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.407314 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.407320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.407326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.407458 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.407849 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.407855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.407858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.408224 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:56.408342 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:56.408751 (+ 409us) server_negotiation.cc:299] Negotiation successful
0621 02:38:56.409132 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":90}
I20260621 02:38:56.414285 7050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: Bootstrap starting.
I20260621 02:38:56.417585 7050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:56.418658 7050 log.cc:824] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:56.421577 7050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: No bootstrap required, opened a new log
I20260621 02:38:56.428347 7050 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } }
I20260621 02:38:56.428694 7050 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:56.428787 7050 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b7d45d78264253924c1d843c95af9a, State: Initialized, Role: FOLLOWER
I20260621 02:38:56.429477 7050 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } }
I20260621 02:38:56.430316 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } } }
I20260621 02:38:56.430671 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:38:56.431351 7050 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:38:56.440030 7067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:38:56.440182 7067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:38:56.442782 6983 master_runner.cc:428] Detected that this master d8b7d45d78264253924c1d843c95af9a is joining an existing cluster
I20260621 02:38:56.442893 6983 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:40413
I20260621 02:38:56.468144 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.452014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.0.97.61:40413)
0621 02:38:56.452215 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:56.452222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:56.452242 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:38:56.452489 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:56.452495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:56.452584 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:56.452654 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:56.454474 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.455753 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.457780 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.458524 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.458627 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.461049 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:56.461056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:56.461060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:56.461092 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:56.465028 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.465877 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.465884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.465888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.466054 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.466484 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.466489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.466492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.466855 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:56.467016 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:56.467485 (+ 469us) server_negotiation.cc:299] Negotiation successful
0621 02:38:56.467756 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 02:38:56.468859 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.451462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.0.1:36902)
0621 02:38:56.452159 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:56.452180 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:56.452320 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:56.452881 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:56.452888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:56.453794 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:56.454292 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.454309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.456059 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.456065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.457643 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.457656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.458930 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.458937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.459121 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.459805 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:56.459828 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:56.460892 (+ 1064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:56.465171 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.465177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.465180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.465749 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.466195 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.466200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.466204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.466368 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.467083 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:56.467087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:56.467241 (+ 154us) client_negotiation.cc:769] Sending connection context
0621 02:38:56.467729 (+ 488us) client_negotiation.cc:240] Negotiation successful
0621 02:38:56.467976 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":600,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260621 02:38:56.471084 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.451336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.0.97.62:46061)
0621 02:38:56.451544 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:56.451551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:56.454688 (+ 3137us) server_negotiation.cc:407] Connection header received
0621 02:38:56.454961 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:56.454967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:56.455043 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:56.455112 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:56.456506 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.457627 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.459977 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.460643 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.460748 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.463888 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:56.463894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:56.463899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:56.463942 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:56.467442 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.468791 (+ 1349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.468798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.468803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.468965 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.469458 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.469463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.469467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.469824 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:56.469936 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:56.470523 (+ 587us) server_negotiation.cc:299] Negotiation successful
0621 02:38:56.470732 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:38:56.472023 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.452599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60168)
0621 02:38:56.454590 (+ 1991us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:56.454608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:56.454781 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:56.455313 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:56.455317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:56.455981 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:56.456326 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.456343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.457810 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.457815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.459809 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.459822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.461010 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.461016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.461216 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.462167 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:56.462189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:56.463668 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:56.467599 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.467603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.467607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.468650 (+ 1043us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.469125 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.469129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.469134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.469322 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.470022 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:56.470026 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:56.470240 (+ 214us) client_negotiation.cc:769] Sending connection context
0621 02:38:56.471339 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0621 02:38:56.471705 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1866,"thread_start_us":1693,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 21 02:38:56 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260621 02:38:56.494851 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.478734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.0.97.61:40413)
0621 02:38:56.478917 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:56.478923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:56.481090 (+ 2167us) server_negotiation.cc:407] Connection header received
0621 02:38:56.481444 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:56.481449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:56.481585 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:56.481709 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:56.483088 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.483812 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.485062 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.485668 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.485798 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.489087 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:56.489094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:56.489100 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:56.489141 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:56.491922 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.492747 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.492755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.492758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.492887 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.493277 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.493310 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.493314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.493574 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:56.493727 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:56.494250 (+ 523us) server_negotiation.cc:299] Negotiation successful
0621 02:38:56.494467 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:38:56.494998 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.478658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.0.1:36908)
0621 02:38:56.480962 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:56.481001 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:56.481224 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:56.481960 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:56.481968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:56.482704 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:56.482902 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.482911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.483941 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.483948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.484907 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.484918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.485831 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.485838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.486019 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.486691 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:56.486708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:56.488893 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:56.492059 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.492072 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.492080 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.492634 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.492997 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.493002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.493006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.493168 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.493708 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:56.493716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:56.493935 (+ 219us) client_negotiation.cc:769] Sending connection context
0621 02:38:56.494164 (+ 229us) client_negotiation.cc:240] Negotiation successful
0621 02:38:56.494467 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2069,"thread_start_us":1913,"threads_started":1}
I20260621 02:38:56.497315 6521 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:40413: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d8b7d45d78264253924c1d843c95af9a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 40413
}
attrs {
promote: true
}
}
dest_uuid: "1522f392977c491790c78c87c51a46b6"
cas_config_opid_index: -1
I20260621 02:38:56.498601 6521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260621 02:38:56.502744 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45537
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46061,127.0.97.61:40413,127.0.97.60:45537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:38:56.513046 7079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d8b7d45d78264253924c1d843c95af9a (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } } }
I20260621 02:38:56.514556 7025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:56.519762 7025 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Refusing update from remote peer 1522f392977c491790c78c87c51a46b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 02:38:56.525642 7080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:56.539863 7079 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:40413
I20260621 02:38:56.540843 6983 master.cc:537] Master@127.0.97.61:40413 shutting down...
I20260621 02:38:56.541816 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } } }
I20260621 02:38:56.542109 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:56.567705 7025 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d8b7d45d78264253924c1d843c95af9a (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } } }
I20260621 02:38:56.573580 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1522f392977c491790c78c87c51a46b6. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } } }
I20260621 02:38:56.573840 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: LEARNER
I20260621 02:38:56.574082 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } } }
I20260621 02:38:56.574301 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: LEARNER
I20260621 02:38:56.574479 7089 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } } }
I20260621 02:38:56.574790 7089 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: LEARNER
I20260621 02:38:56.599779 6983 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 LEARNER]: Raft consensus shutting down.
I20260621 02:38:56.600346 6983 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 LEARNER]: Raft consensus is shut down!
I20260621 02:38:56.600521 6983 tablet_replica.cc:330] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: stopping tablet replica
I20260621 02:38:56.609112 6983 master.cc:552] Master@127.0.97.61:40413 shutdown complete.
I20260621 02:38:56.609248 6983 master_runner.cc:305] Clearing existing system tablet
I20260621 02:38:56.609730 6983 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:56.609862 6983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:56.612782 6983 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:38:56.617219 7091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:56.617587 6983 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:56.618021 6983 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260621 02:38:56.618150 6983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal
uuid: "d8b7d45d78264253924c1d843c95af9a"
format_stamp: "Formatted at 2026-06-21 02:38:56 on dist-test-slave-jkn7"
server_key: "8a63e6c24c3f81ab92f256921e25bf32"
server_key_iv: "3e70cb55358b898d5d5892c113cb4bb2"
server_key_version: "encryptionkey@0"
I20260621 02:38:56.623075 6983 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:56.630309 6983 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 02:38:56.630555 6983 log.cc:1196] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 02:38:56.631199 6983 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: Deleting consensus metadata
I20260621 02:38:56.631502 6983 master_runner.cc:315] Copying system tablet from 127.0.97.62:46061
I20260621 02:38:56.634240 6983 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46061
I20260621 02:38:56.650154 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.635185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.0.97.62:46061)
0621 02:38:56.635419 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:56.635426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:56.636041 (+ 615us) server_negotiation.cc:407] Connection header received
0621 02:38:56.636257 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:56.636262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:56.636353 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:56.636463 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:56.637510 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.638606 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.640383 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.641016 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.641147 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.643548 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:56.643554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:56.643559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:56.643598 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:56.647038 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.647933 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.647938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.647942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.648073 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:56.648523 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:56.648528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:56.648532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:56.648897 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:56.649046 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:56.649548 (+ 502us) server_negotiation.cc:299] Negotiation successful
0621 02:38:56.649780 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:38:56.650197 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:56.635103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60182)
0621 02:38:56.635887 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:56.635925 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:56.636107 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:56.636512 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:56.636517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:56.637019 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:56.637344 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.637358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.638758 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.638763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.640235 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:56.640248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:56.641229 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:56.641235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:56.641512 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:56.642238 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:56.642259 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:56.643345 (+ 1086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:56.647212 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.647220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.647226 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.647800 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.648207 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:56.648215 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:56.648219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:56.648394 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:56.649037 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:56.649041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:56.649233 (+ 192us) client_negotiation.cc:769] Sending connection context
0621 02:38:56.649444 (+ 211us) client_negotiation.cc:240] Negotiation successful
0621 02:38:56.649747 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":479,"threads_started":1}
I20260621 02:38:56.651358 6542 tablet_copy_service.cc:140] P 1522f392977c491790c78c87c51a46b6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d8b7d45d78264253924c1d843c95af9a ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60182)
I20260621 02:38:56.651621 6542 tablet_copy_service.cc:161] P 1522f392977c491790c78c87c51a46b6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d8b7d45d78264253924c1d843c95af9a at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60182: session id = d8b7d45d78264253924c1d843c95af9a-00000000000000000000000000000000
I20260621 02:38:56.656015 6542 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 02:38:56.658237 6983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 02:38:56.665058 6983 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: tablet copy: Starting download of 0 data blocks...
I20260621 02:38:56.665508 6983 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: tablet copy: Starting download of 1 WAL segments...
I20260621 02:38:56.668406 6983 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 02:38:56.672689 6542 tablet_copy_service.cc:342] P 1522f392977c491790c78c87c51a46b6: Request end of tablet copy session d8b7d45d78264253924c1d843c95af9a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60182
I20260621 02:38:56.672889 6542 tablet_copy_service.cc:433] P 1522f392977c491790c78c87c51a46b6: ending tablet copy session d8b7d45d78264253924c1d843c95af9a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d8b7d45d78264253924c1d843c95af9a
I20260621 02:38:56.684386 6983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:56.687224 7098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:56.692328 7099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:56.699419 7101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:56.699822 6983 server_base.cc:1034] running on GCE node
I20260621 02:38:56.700325 6983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:56.701231 6983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:56.702585 6983 hybrid_clock.cc:648] HybridClock initialized: now 1782009536702504 us; error 56 us; skew 500 ppm
I20260621 02:38:56.702983 6983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:56 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009536, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:56.705523 6983 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260621 02:38:56.713485 6983 webserver.cc:460] Webserver started at http://127.0.97.61:33071/ using document root <none> and password file <none>
I20260621 02:38:56.713935 6983 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:56.714116 6983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:56.717995 6983 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260621 02:38:56.720269 7108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:56.720788 6983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260621 02:38:56.720966 6983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal
uuid: "d8b7d45d78264253924c1d843c95af9a"
format_stamp: "Formatted at 2026-06-21 02:38:56 on dist-test-slave-jkn7"
server_key: "8a63e6c24c3f81ab92f256921e25bf32"
server_key_iv: "3e70cb55358b898d5d5892c113cb4bb2"
server_key_version: "encryptionkey@0"
I20260621 02:38:56.721223 6983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:56.735077 6983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:56.739862 6983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:56.764138 7159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40413 every 8 connection(s)
I20260621 02:38:56.764361 6983 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40413
I20260621 02:38:56.765347 6983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
I20260621 02:38:56.774628 7160 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:38:56.779893 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: Bootstrap starting.
I20260621 02:38:56.797675 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:38:56.798215 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: Bootstrap complete.
I20260621 02:38:56.799149 7160 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } }
I20260621 02:38:56.799371 7160 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8b7d45d78264253924c1d843c95af9a, State: Initialized, Role: LEARNER
I20260621 02:38:56.799605 7160 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } }
I20260621 02:38:56.800781 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } } }
I20260621 02:38:56.801148 7160 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:56.801301 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: LEARNER
I20260621 02:38:56.820283 7174 catalog_manager.cc:1259] Loaded cluster ID: 0c06c1c544434673a79cfe1a0e53415f
I20260621 02:38:56.820479 7174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: loading cluster ID for follower catalog manager: success
I20260621 02:38:56.823367 7174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: acquiring CA information for follower catalog manager: success
I20260621 02:38:56.825158 7174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:38:56.892293 6978 heartbeater.cc:503] Master 127.0.97.62:46061 was elected leader, sending a full tablet report...
W20260621 02:38:56.963131 7081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:56.963439 7081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:56.963603 7081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:56.970975 7081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:38:56.971109 7081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:56.971215 7081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:56.971313 7081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:38:56.971387 7081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:38:56.981235 7081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46061,127.0.97.61:40413,127.0.97.60:45537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:56.982568 7081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:56.983834 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:56.996166 7179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:56.996210 7182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:56.996248 7180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:56.997253 7081 server_base.cc:1034] running on GCE node
I20260621 02:38:56.998011 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:56.999423 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:57.000613 7081 hybrid_clock.cc:648] HybridClock initialized: now 1782009537000562 us; error 57 us; skew 500 ppm
I20260621 02:38:57.000960 7081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:57.010684 7081 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260621 02:38:57.012792 7081 webserver.cc:460] Webserver started at http://127.0.97.60:34741/ using document root <none> and password file <none>
I20260621 02:38:57.013651 7081 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:57.013790 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:57.014128 7081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:38:57.017135 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } } }
I20260621 02:38:57.017439 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:57.017549 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data/instance:
uuid: "c5dcd042c98242c89b56db2487519736"
format_stamp: "Formatted at 2026-06-21 02:38:57 on dist-test-slave-jkn7"
server_key: "c9da08cd556e23ed486e57c8bcbce333"
server_key_iv: "f02bcc2eabde64c2cbd8dc83dbc14114"
server_key_version: "encryptionkey@0"
I20260621 02:38:57.018661 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal/instance:
uuid: "c5dcd042c98242c89b56db2487519736"
format_stamp: "Formatted at 2026-06-21 02:38:57 on dist-test-slave-jkn7"
server_key: "c9da08cd556e23ed486e57c8bcbce333"
server_key_iv: "f02bcc2eabde64c2cbd8dc83dbc14114"
server_key_version: "encryptionkey@0"
I20260621 02:38:57.024931 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.017529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.0.1:36920)
0621 02:38:57.018252 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.018273 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.018411 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.018931 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.018935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.018958 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:57.019137 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.019144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.020930 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.020939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.022540 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.022554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.024127 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.024133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.024343 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.024424 (+ 81us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.024541 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.024690 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":275,"threads_started":1}
I20260621 02:38:57.025365 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.017665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.0.97.61:40413)
0621 02:38:57.018488 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.018494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.018517 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:38:57.018608 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.018616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.018723 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.018860 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:57.019282 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.020765 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.022680 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.023957 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.024065 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.024238 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.024656 (+ 418us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.024804 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":523,"threads_started":1}
I20260621 02:38:57.025597 7081 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260621 02:38:57.026343 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1522f392977c491790c78c87c51a46b6. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: true } } }
I20260621 02:38:57.026650 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: LEARNER
I20260621 02:38:57.027659 7079 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: attempting to promote NON_VOTER d8b7d45d78264253924c1d843c95af9a to VOTER
I20260621 02:38:57.028486 7079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } }
I20260621 02:38:57.030153 7191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:57.030395 7135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 LEARNER]: Refusing update from remote peer 1522f392977c491790c78c87c51a46b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 02:38:57.031047 7080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 02:38:57.032142 7081 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:38:57.032328 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal
uuid: "c5dcd042c98242c89b56db2487519736"
format_stamp: "Formatted at 2026-06-21 02:38:57 on dist-test-slave-jkn7"
server_key: "c9da08cd556e23ed486e57c8bcbce333"
server_key_iv: "f02bcc2eabde64c2cbd8dc83dbc14114"
server_key_version: "encryptionkey@0"
I20260621 02:38:57.032532 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:57.034058 7079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8b7d45d78264253924c1d843c95af9a (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } }
I20260621 02:38:57.034755 7135 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8b7d45d78264253924c1d843c95af9a (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } }
I20260621 02:38:57.036590 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } }
I20260621 02:38:57.036825 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:57.036564 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } }
I20260621 02:38:57.036971 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:57.039971 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } }
I20260621 02:38:57.040217 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:38:57.067670 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:57.071762 7081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:57.072038 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:57.091384 7081 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45537
I20260621 02:38:57.091426 7243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45537 every 8 connection(s)
I20260621 02:38:57.092672 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
I20260621 02:38:57.096426 7244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:57.099822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 7081
I20260621 02:38:57.104334 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260621 02:38:57.123893 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.108152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.0.97.62:46061)
0621 02:38:57.108378 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.108384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.108625 (+ 241us) server_negotiation.cc:407] Connection header received
0621 02:38:57.108798 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.108804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.108887 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.108996 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.110152 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.110871 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.112253 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.112672 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.112812 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.116779 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.116787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.116794 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.116833 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.120102 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.121060 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.121064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.121083 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.121235 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.121800 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.121804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.121839 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.122193 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.122346 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.123210 (+ 864us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.123426 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260621 02:38:57.124955 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.102044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.0.97.60:45537)
0621 02:38:57.102859 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.102870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.102908 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:38:57.104602 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.104623 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.105058 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.105447 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.107313 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.109954 (+ 2641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.111294 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.112081 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.112263 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.117704 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.117716 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.117733 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.117769 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.121022 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.121761 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.121769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.121775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.121907 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.122313 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.122317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.122320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.122757 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.123043 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.123372 (+ 329us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.123699 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":628,"thread_start_us":395,"threads_started":1}
I20260621 02:38:57.124964 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.107391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60190)
0621 02:38:57.108489 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.108534 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.108676 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.109041 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.109046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.109751 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.110008 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.110017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.111009 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.111017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.112123 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.112133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.112869 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.112874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.113119 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.113685 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.113717 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.116593 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.120227 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.120235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.120246 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.120951 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.121365 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.121370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.121382 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.121655 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.122321 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.122329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.122765 (+ 436us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.123110 (+ 345us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.123609 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":49,"spinlock_wait_cycles":17792,"thread_start_us":744,"threads_started":1}
I20260621 02:38:57.126837 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260621 02:38:57.138893 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.127556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.0.1:36922)
0621 02:38:57.127731 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.127750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.127854 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.128169 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.128172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.128449 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.128637 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.128652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.129685 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.129689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.130567 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.130577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.131221 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.131226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.131371 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.131855 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.131873 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.133893 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.136628 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.136632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.136635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.136995 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.137479 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.137482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.137484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.137612 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.138124 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.138127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.138240 (+ 113us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.138401 (+ 161us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.138658 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 02:38:57.139410 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.127604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.0.97.61:40413)
0621 02:38:57.127779 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.127793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.127809 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:38:57.127980 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.127985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.128064 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.128149 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.128771 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.129562 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.130690 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.131087 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.131178 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.134093 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.134097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.134101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.134131 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.136501 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.137144 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.137151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.137154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.137350 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.137703 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.137706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.137708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.138013 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.138135 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.138496 (+ 361us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.138681 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:38:57.140385 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260621 02:38:57.151979 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.141005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45537 (local address 127.0.0.1:55186)
0621 02:38:57.141190 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.141210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.141365 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.141672 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.141676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.142041 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.142170 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.142178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.143125 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.143133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.144010 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.144020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.144708 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.144714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.144915 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.145454 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.145470 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.147288 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.149817 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.149824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.149828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.150142 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.150481 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.150486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.150489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.150622 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.151076 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.151081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.151205 (+ 124us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.151332 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.151580 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:38:57.152009 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.141038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.0.97.60:45537)
0621 02:38:57.141191 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.141198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.141307 (+ 109us) server_negotiation.cc:407] Connection header received
0621 02:38:57.141493 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.141505 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.141558 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.141658 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.142301 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.143002 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.144147 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.144567 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.144704 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.147464 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.147469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.147472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.147496 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.149700 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.150249 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.150256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.150260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.150370 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.150731 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.150736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.150739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.150964 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.151110 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.151400 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.151698 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":46}
I20260621 02:38:57.156457 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Bootstrap starting.
I20260621 02:38:57.159819 7244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:57.160728 7244 log.cc:824] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:57.163216 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: No bootstrap required, opened a new log
I20260621 02:38:57.169842 7244 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } }
I20260621 02:38:57.170120 7244 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:57.170204 7244 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5dcd042c98242c89b56db2487519736, State: Initialized, Role: FOLLOWER
I20260621 02:38:57.170663 7244 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } }
I20260621 02:38:57.171360 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } } }
I20260621 02:38:57.171693 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:38:57.172281 7244 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:57.179953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 6583
W20260621 02:38:57.181260 7261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:38:57.181412 7261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:38:57.185603 7081 master_runner.cc:428] Detected that this master c5dcd042c98242c89b56db2487519736 is joining an existing cluster
I20260621 02:38:57.185710 7081 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:45537
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260621 02:38:57.203712 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.181598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.0.97.1:38313)
0621 02:38:57.182186 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.182194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.182223 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:38:57.182286 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.182295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.182531 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.182686 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.184344 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.185785 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.187705 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.188536 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.188633 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.194469 (+ 5836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.194483 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.194497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.194550 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.198506 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.199565 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.199574 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.199577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.199736 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.200702 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.200707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.200710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.201200 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.201511 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.202973 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.203245 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":285,"threads_started":1}
I20260621 02:38:57.205152 6706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:57.222337 7250 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:57.222648 7250 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } }
I20260621 02:38:57.224354 7250 leader_election.cc:288] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1522f392977c491790c78c87c51a46b6 (127.0.97.62:46061), d8b7d45d78264253924c1d843c95af9a (127.0.97.61:40413)
I20260621 02:38:57.225435 7135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5dcd042c98242c89b56db2487519736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b7d45d78264253924c1d843c95af9a" is_pre_election: true
I20260621 02:38:57.225981 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.189453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.0.97.61:40413)
0621 02:38:57.189657 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.189665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.198092 (+ 8427us) server_negotiation.cc:407] Connection header received
0621 02:38:57.199015 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.199021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.199117 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.199189 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.206276 (+ 7087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.207652 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.212627 (+ 4975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.213557 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.213685 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.218966 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.218972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.218977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.219019 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.222661 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.223508 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.223514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.223518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.223676 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.224148 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.224153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.224157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.224507 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.224647 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.225243 (+ 596us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.225612 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:38:57.226750 6532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5dcd042c98242c89b56db2487519736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1522f392977c491790c78c87c51a46b6" is_pre_election: true
I20260621 02:38:57.227212 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.189394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.0.1:36926)
0621 02:38:57.197976 (+ 8582us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.197997 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.198415 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.199546 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.199551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.200703 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.201083 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.201096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.207907 (+ 6811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.207916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.210056 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.210076 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.214068 (+ 3992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.214077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.214309 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.217318 (+ 3009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.217349 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.218763 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.222870 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.222875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.222879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.223364 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.223824 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.223828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.223832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.224015 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.224719 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.224724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.224916 (+ 192us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.226410 (+ 1494us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.226739 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8035,"thread_start_us":7379,"threads_started":1}
I20260621 02:38:57.228132 7196 leader_election.cc:302] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5dcd042c98242c89b56db2487519736; no voters: 1522f392977c491790c78c87c51a46b6, d8b7d45d78264253924c1d843c95af9a
I20260621 02:38:57.228834 7250 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 02:38:57.231873 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.193677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.0.97.62:46061)
0621 02:38:57.193855 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.193861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.198898 (+ 5037us) server_negotiation.cc:407] Connection header received
0621 02:38:57.201308 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.201315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.201418 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.201487 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.202938 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.204007 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.210271 (+ 6264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.211203 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.211326 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.220013 (+ 8687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.220020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.220036 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.220076 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.223817 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.229183 (+ 5366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.229192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.229199 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.229401 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.229850 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.229855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.229859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.230183 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.230305 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.230892 (+ 587us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.231129 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:38:57.232920 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.194044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60196)
0621 02:38:57.198804 (+ 4760us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.198821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.199377 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.201699 (+ 2322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.201703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.202384 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.202764 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.202779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.204201 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.204206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.206048 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.206068 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.211669 (+ 5601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.211677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.211936 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.215522 (+ 3586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.215555 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.218976 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.223976 (+ 5000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.223981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.223984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.224454 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.229550 (+ 5096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.229555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.229559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.229725 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.230391 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.230399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.230620 (+ 221us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.232103 (+ 1483us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.232433 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4625,"spinlock_wait_cycles":35584,"thread_start_us":4190,"threads_started":1}
I20260621 02:38:57.241385 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.198947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45537 (local address 127.0.0.1:55188)
0621 02:38:57.227798 (+ 28851us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.227817 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.227951 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.228384 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.228389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.228912 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.229083 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.229102 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.230489 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.230494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.232208 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.232220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.233498 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.233505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.233676 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.234295 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.234310 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.235120 (+ 810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.239079 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.239086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.239090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.239415 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.239769 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.239774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.239777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.239883 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.240399 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.240404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.240516 (+ 112us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.240680 (+ 164us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.240958 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28751,"thread_start_us":34237,"threads_started":1}
I20260621 02:38:57.241438 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.198750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.0.97.60:45537)
0621 02:38:57.198917 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.198923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.227873 (+ 28950us) server_negotiation.cc:407] Connection header received
0621 02:38:57.228086 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.228091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.228160 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.228219 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.229217 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.230300 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.232350 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.233087 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.233192 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.235277 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.235283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.235287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.235319 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.238965 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.239532 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.239538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.239541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.239663 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.239978 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.239985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.239989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.240248 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.240398 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.240763 (+ 365us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.241083 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":38}
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260621 02:38:57.256186 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.238520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45537 (local address 127.0.0.1:55192)
0621 02:38:57.238697 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.238717 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.238815 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.242260 (+ 3445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.242264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.242687 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.242871 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.242882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.243966 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.243972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.244835 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.244845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.245712 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.245718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.245902 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.246521 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.246545 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.249048 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.253516 (+ 4468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.253521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.253526 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.254002 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.254421 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.254426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.254430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.254567 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.255187 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.255193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.255386 (+ 193us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.255533 (+ 147us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.255791 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 02:38:57.257205 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.238845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.0.97.60:45537)
0621 02:38:57.241663 (+ 2818us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.241668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.241684 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:38:57.241754 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.241758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.241858 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.241955 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.243018 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.243835 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.244967 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.245539 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.249220 (+ 3681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.249851 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.249858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.249869 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.249902 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.253359 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.254165 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.254170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.254174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.254288 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.254701 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.254706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.254709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.255041 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.256522 (+ 1481us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.256631 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.256846 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2725,"thread_start_us":10970,"threads_started":1}
I20260621 02:38:57.258623 6521 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:45537: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c5dcd042c98242c89b56db2487519736"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 45537
}
attrs {
promote: true
}
}
dest_uuid: "1522f392977c491790c78c87c51a46b6"
cas_config_opid_index: 6
I20260621 02:38:57.259339 6521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } }
I20260621 02:38:57.261471 7135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Refusing update from remote peer 1522f392977c491790c78c87c51a46b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260621 02:38:57.262151 7219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:57.264067 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260621 02:38:57.266539 7080 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5dcd042c98242c89b56db2487519736 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } } }
I20260621 02:38:57.267074 7135 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5dcd042c98242c89b56db2487519736 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } } }
I20260621 02:38:57.270431 7080 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:45537
I20260621 02:38:57.270795 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } } }
I20260621 02:38:57.270849 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } } }
I20260621 02:38:57.271113 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:57.271152 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:57.270578 7219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 FOLLOWER]: Refusing update from remote peer 1522f392977c491790c78c87c51a46b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260621 02:38:57.271880 7081 master.cc:537] Master@127.0.97.60:45537 shutting down...
I20260621 02:38:57.272899 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } } }
I20260621 02:38:57.273169 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:38:57.273988 7080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:38:57.302124 6509 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260621 02:38:57.303962 7081 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:57.304352 7081 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 02:38:57.304302 6509 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 -> Peer c5dcd042c98242c89b56db2487519736 (127.0.97.60:45537): Couldn't send request to peer c5dcd042c98242c89b56db2487519736. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 02:38:57.304562 7081 tablet_replica.cc:330] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: stopping tablet replica
I20260621 02:38:57.311326 7081 master.cc:552] Master@127.0.97.60:45537 shutdown complete.
I20260621 02:38:57.311476 7081 master_runner.cc:305] Clearing existing system tablet
I20260621 02:38:57.311936 7081 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:57.312093 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:57.314952 7081 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260621 02:38:57.316640 7275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:57.317039 7081 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:57.317480 7081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260621 02:38:57.317627 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal
uuid: "c5dcd042c98242c89b56db2487519736"
format_stamp: "Formatted at 2026-06-21 02:38:57 on dist-test-slave-jkn7"
server_key: "c9da08cd556e23ed486e57c8bcbce333"
server_key_iv: "f02bcc2eabde64c2cbd8dc83dbc14114"
server_key_version: "encryptionkey@0"
I20260621 02:38:57.323194 7081 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:57.330224 7081 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 02:38:57.330475 7081 log.cc:1196] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 02:38:57.331034 7081 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Deleting consensus metadata
I20260621 02:38:57.331341 7081 master_runner.cc:315] Copying system tablet from 127.0.97.62:46061
I20260621 02:38:57.333971 7081 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46061
I20260621 02:38:57.338891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 6583
I20260621 02:38:57.353118 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.334917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.0.97.62:46061)
0621 02:38:57.335155 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.335161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.338074 (+ 2913us) server_negotiation.cc:407] Connection header received
0621 02:38:57.338262 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.338268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.338347 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.338452 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.339656 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.340908 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.342810 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.343590 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.343730 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.345973 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.345982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.345991 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.346033 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.349865 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.350640 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.350648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.350654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.350807 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.351346 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.351354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.351361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.351733 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.351886 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.352487 (+ 601us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.352715 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260621 02:38:57.353349 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.334943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60198)
0621 02:38:57.337941 (+ 2998us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.337962 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.338129 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.338495 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.338500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.339009 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.339489 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.339505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.341075 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.341082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.342649 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.342665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.343762 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.343768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.343947 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.344663 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.344692 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.345779 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.350001 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.350007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.350013 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.350501 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.350930 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.350935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.350939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.351112 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.351954 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.351962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.352180 (+ 218us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.352334 (+ 154us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.352739 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2857,"thread_start_us":2722,"threads_started":1}
I20260621 02:38:57.357699 6542 tablet_copy_service.cc:140] P 1522f392977c491790c78c87c51a46b6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c5dcd042c98242c89b56db2487519736 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60198)
I20260621 02:38:57.357888 6542 tablet_copy_service.cc:161] P 1522f392977c491790c78c87c51a46b6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c5dcd042c98242c89b56db2487519736 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60198: session id = c5dcd042c98242c89b56db2487519736-00000000000000000000000000000000
I20260621 02:38:57.362027 6542 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 02:38:57.362731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38313
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46061
--webserver_port=32857
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46061,127.0.97.61:40413
--tserver_master_addrs=127.0.97.62:46061,127.0.97.61:40413,127.0.97.60:45537 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:38:57.363910 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 02:38:57.370646 7081 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: tablet copy: Starting download of 0 data blocks...
I20260621 02:38:57.370932 7081 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: tablet copy: Starting download of 1 WAL segments...
I20260621 02:38:57.373368 7081 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 02:38:57.381541 6542 tablet_copy_service.cc:342] P 1522f392977c491790c78c87c51a46b6: Request end of tablet copy session c5dcd042c98242c89b56db2487519736-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60198
I20260621 02:38:57.381687 6542 tablet_copy_service.cc:433] P 1522f392977c491790c78c87c51a46b6: ending tablet copy session c5dcd042c98242c89b56db2487519736-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c5dcd042c98242c89b56db2487519736
I20260621 02:38:57.394196 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:57.396716 7283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:57.397975 7284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:57.405203 7286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:57.405589 7081 server_base.cc:1034] running on GCE node
I20260621 02:38:57.405989 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:57.406823 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:57.408138 7081 hybrid_clock.cc:648] HybridClock initialized: now 1782009537408098 us; error 33 us; skew 500 ppm
I20260621 02:38:57.408367 7081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:57.411082 7081 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260621 02:38:57.425632 7081 webserver.cc:460] Webserver started at http://127.0.97.60:43239/ using document root <none> and password file <none>
I20260621 02:38:57.425982 7081 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:57.426100 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:57.436944 7081 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.008s sys 0.003s
I20260621 02:38:57.445694 7293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:57.446139 7081 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260621 02:38:57.446338 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal
uuid: "c5dcd042c98242c89b56db2487519736"
format_stamp: "Formatted at 2026-06-21 02:38:57 on dist-test-slave-jkn7"
server_key: "c9da08cd556e23ed486e57c8bcbce333"
server_key_iv: "f02bcc2eabde64c2cbd8dc83dbc14114"
server_key_version: "encryptionkey@0"
I20260621 02:38:57.446576 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:57.464624 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:57.467995 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:57.495661 7081 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45537
I20260621 02:38:57.495720 7344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45537 every 8 connection(s)
I20260621 02:38:57.496695 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
I20260621 02:38:57.501798 7345 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:38:57.505833 7345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Bootstrap starting.
I20260621 02:38:57.523684 7345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:38:57.524214 7345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Bootstrap complete.
I20260621 02:38:57.525374 7345 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } }
I20260621 02:38:57.525749 7345 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5dcd042c98242c89b56db2487519736, State: Initialized, Role: LEARNER
I20260621 02:38:57.526108 7345 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } }
I20260621 02:38:57.529753 7348 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } } }
I20260621 02:38:57.530061 7345 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:38:57.530179 7348 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:38:57.549712 7359 catalog_manager.cc:1259] Loaded cluster ID: 0c06c1c544434673a79cfe1a0e53415f
I20260621 02:38:57.549834 7359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: loading cluster ID for follower catalog manager: success
I20260621 02:38:57.553267 7359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: acquiring CA information for follower catalog manager: success
I20260621 02:38:57.555583 7359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:38:57.748030 7282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:57.748333 7282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:57.748503 7282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:57.755698 7282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:57.755842 7282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:57.756022 7282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:38:57.765584 7282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=32857
--tserver_master_addrs=127.0.97.62:46061,127.0.97.61:40413,127.0.97.60:45537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:57.766803 7282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:57.768071 7282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:57.783080 7364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:57.783105 7365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:57.783279 7367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:57.783838 7282 server_base.cc:1034] running on GCE node
I20260621 02:38:57.784417 7282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:57.785435 7282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:57.786669 7282 hybrid_clock.cc:648] HybridClock initialized: now 1782009537786618 us; error 41 us; skew 500 ppm
I20260621 02:38:57.786994 7282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:57.790318 7282 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260621 02:38:57.791823 7282 webserver.cc:460] Webserver started at http://127.0.97.1:32857/ using document root <none> and password file <none>
I20260621 02:38:57.792508 7282 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:57.792619 7282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:57.797858 7282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:38:57.800559 7374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:57.802131 7282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:57.802296 7282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "406a3d74c2044d018ddd85d9bbff38ca"
format_stamp: "Formatted at 2026-06-21 02:38:54 on dist-test-slave-jkn7"
server_key: "eefd6b0736f7d3bc28e18d1891f73ea1"
server_key_iv: "322c1c9c746a491c1c796b2f9b5e3c33"
server_key_version: "encryptionkey@0"
I20260621 02:38:57.803328 7282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:57.826326 7282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:57.830646 7282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:57.830924 7282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:57.831881 7282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:57.833212 7282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:57.833352 7282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:57.833467 7282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:57.833568 7282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:57.871289 7282 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38313
I20260621 02:38:57.871336 7488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38313 every 8 connection(s)
I20260621 02:38:57.872687 7282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:38:57.876240 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.867526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45537 (local address 127.0.0.1:55202)
0621 02:38:57.868070 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.868099 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.868267 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.870120 (+ 1853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.870124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.870147 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:57.870299 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.870306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.871713 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.871719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.873035 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.873047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.875156 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.875166 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.875441 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.875497 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.875585 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.875783 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":218,"threads_started":1}
I20260621 02:38:57.876891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 7282
I20260621 02:38:57.877247 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 6717
I20260621 02:38:57.878085 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.867791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.0.97.60:45537)
0621 02:38:57.869724 (+ 1933us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.869731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.869755 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:38:57.869831 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.869836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.869928 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.870421 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:57.870569 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.871572 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.873240 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.874906 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.876899 (+ 1993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.877259 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.877421 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.877636 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1778,"thread_start_us":1672,"threads_started":1}
I20260621 02:38:57.879454 7320 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260621 02:38:57.879994 7348 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1522f392977c491790c78c87c51a46b6. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } } }
I20260621 02:38:57.880273 7348 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: This master's current role is: LEARNER
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 21 02:38:57 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260621 02:38:57.918366 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.878754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.0.97.2:45449)
0621 02:38:57.891374 (+ 12620us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.891389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.892135 (+ 746us) server_negotiation.cc:407] Connection header received
0621 02:38:57.892315 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.892328 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.892664 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.892844 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.895349 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.896841 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.898844 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.899675 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.899803 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.906685 (+ 6882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.906703 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.906724 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.906777 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.911109 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.913882 (+ 2773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.913895 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.913899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.914819 (+ 920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.915424 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.915430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.915436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.915967 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.916300 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.917579 (+ 1279us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.917879 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12371,"thread_start_us":12185,"threads_started":1}
I20260621 02:38:57.919979 6840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:57.923408 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.893961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50143 (local address 127.0.97.60:45537)
0621 02:38:57.894152 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.894157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.896542 (+ 2385us) server_negotiation.cc:407] Connection header received
0621 02:38:57.898028 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.898036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.898115 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.898205 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.900693 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.901817 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.904767 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.905467 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.905602 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.913963 (+ 8361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.913971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.913976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.914023 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.917194 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.920280 (+ 3086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.920290 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.920296 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.920497 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.921060 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.921067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.921073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.921574 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.921752 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.922910 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.923146 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:38:57.925676 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.893840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45537 (local address 127.0.97.1:50143)
0621 02:38:57.896409 (+ 2569us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.896431 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.897884 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.898517 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.898529 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.899534 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.900521 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.900529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.902108 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.902112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.903920 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.903935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.905985 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.905991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.906218 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.907701 (+ 1483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.907725 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.913714 (+ 5989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.917569 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.917574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.917578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.918364 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.920706 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.920713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.920719 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.920906 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.921875 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.921887 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.922459 (+ 572us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.922845 (+ 386us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.923567 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2426,"mutex_wait_us":170,"spinlock_wait_cycles":14464,"thread_start_us":2278,"threads_started":1}
I20260621 02:38:57.926029 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.876709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39243 (local address 127.0.97.62:46061)
0621 02:38:57.894496 (+ 17787us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.894504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.895274 (+ 770us) server_negotiation.cc:407] Connection header received
0621 02:38:57.898057 (+ 2783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.898063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.898131 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.898224 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.900688 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.902362 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.904462 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.905096 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.905223 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.913395 (+ 8172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.913404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.913409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.913456 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.917141 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.918422 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.918427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.918430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.918601 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.919104 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.919109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.919113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.919480 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.919638 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.925379 (+ 5741us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.925629 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17655,"thread_start_us":17777,"threads_started":1}
I20260621 02:38:57.926733 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.876253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.97.1:49281)
0621 02:38:57.892921 (+ 16668us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.893010 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.897548 (+ 4538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.898517 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.898529 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.899534 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.900484 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.900513 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.902160 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.902165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.905097 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.905109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.906369 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.906375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.906574 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.907357 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.907382 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.912951 (+ 5569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.917608 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.917613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.917620 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.918393 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.918873 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.918878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.918881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.919078 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.925758 (+ 6680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.925763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.925893 (+ 130us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.926068 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.926346 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16288,"mutex_wait_us":1493,"spinlock_wait_cycles":77440,"thread_start_us":18276,"threads_started":1}
I20260621 02:38:57.927448 7489 heartbeater.cc:347] Connected to a master server at 127.0.97.60:45537
I20260621 02:38:57.927701 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.877147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49281 (local address 127.0.97.61:40413)
0621 02:38:57.893642 (+ 16495us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:57.893651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:57.894950 (+ 1299us) server_negotiation.cc:407] Connection header received
0621 02:38:57.897814 (+ 2864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:57.897819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:57.897925 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:57.898053 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:57.900719 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.901947 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.905249 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.905917 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.906034 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.913338 (+ 7304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:57.913346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:57.913351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:57.913398 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:57.917434 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.918558 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.918564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.918568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.918735 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:57.919180 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:57.919189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:57.919195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:57.919501 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:57.919638 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:57.927118 (+ 7480us) server_negotiation.cc:299] Negotiation successful
0621 02:38:57.927345 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16355,"thread_start_us":16270,"threads_started":1}
I20260621 02:38:57.927758 7489 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:57.928563 7491 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40413
I20260621 02:38:57.928630 7489 heartbeater.cc:511] Master 127.0.97.60:45537 requested a full tablet report, sending...
I20260621 02:38:57.928702 7491 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:57.929175 7491 heartbeater.cc:511] Master 127.0.97.61:40413 requested a full tablet report, sending...
I20260621 02:38:57.929450 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:57.876253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.1:39243)
0621 02:38:57.894969 (+ 18716us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:57.895002 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:57.897492 (+ 2490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:57.898517 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:57.898529 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:57.899426 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:57.900480 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.900511 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.902558 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.902565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.904300 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:57.904316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:57.905531 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:57.905537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:57.905884 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:57.907244 (+ 1360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:57.907282 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:57.913059 (+ 5777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:57.917379 (+ 4320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.917391 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.917410 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.918246 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.918730 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:57.918735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:57.918738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:57.918968 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:57.924386 (+ 5418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:57.924391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:57.925004 (+ 613us) client_negotiation.cc:769] Sending connection context
0621 02:38:57.925146 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 02:38:57.925454 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18568,"mutex_wait_us":3641,"thread_start_us":12578,"threads_started":1}
I20260621 02:38:57.931237 7125 ts_manager.cc:194] Registered new tserver with Master: 406a3d74c2044d018ddd85d9bbff38ca (127.0.97.1:38313)
I20260621 02:38:57.931233 7310 ts_manager.cc:194] Registered new tserver with Master: 406a3d74c2044d018ddd85d9bbff38ca (127.0.97.1:38313)
I20260621 02:38:57.931717 7490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46061
I20260621 02:38:57.931846 7490 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:57.932294 7490 heartbeater.cc:511] Master 127.0.97.62:46061 requested a full tablet report, sending...
I20260621 02:38:57.933372 6521 ts_manager.cc:194] Re-registered known tserver with Master: 406a3d74c2044d018ddd85d9bbff38ca (127.0.97.1:38313)
I20260621 02:38:57.934149 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39243
I20260621 02:38:58.006695 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 6717
I20260621 02:38:58.024621 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45449
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46061
--webserver_port=39673
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46061,127.0.97.61:40413
--tserver_master_addrs=127.0.97.62:46061,127.0.97.61:40413,127.0.97.60:45537 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:38:58.313990 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: true } } }
I20260621 02:38:58.314404 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:58.315364 7080 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: attempting to promote NON_VOTER c5dcd042c98242c89b56db2487519736 to VOTER
I20260621 02:38:58.315984 7080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } }
I20260621 02:38:58.322816 7320 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 LEARNER]: Refusing update from remote peer 1522f392977c491790c78c87c51a46b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 02:38:58.322863 7135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Refusing update from remote peer 1522f392977c491790c78c87c51a46b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 02:38:58.323482 7080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 02:38:58.323877 7503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 02:38:58.329913 7502 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5dcd042c98242c89b56db2487519736 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:38:58.330353 7320 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5dcd042c98242c89b56db2487519736 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:38:58.333869 7502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:38:58.334347 7502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:58.334652 7502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:38:58.334451 7135 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5dcd042c98242c89b56db2487519736 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:38:58.334951 7502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:58.337474 7348 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:38:58.337733 7348 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:38:58.338219 7503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:38:58.338481 7503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:38:58.338335 7509 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1522f392977c491790c78c87c51a46b6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:38:58.338699 7509 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: FOLLOWER
W20260621 02:38:58.355875 7501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:58.356124 7501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:58.356215 7501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:58.363253 7501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:58.363348 7501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:58.363456 7501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:38:58.372835 7501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39673
--tserver_master_addrs=127.0.97.62:46061,127.0.97.61:40413,127.0.97.60:45537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:58.374069 7501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:58.375240 7501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:58.388083 7514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:58.387990 7517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:58.388019 7515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:58.388823 7501 server_base.cc:1034] running on GCE node
I20260621 02:38:58.389490 7501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:58.390398 7501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:58.391618 7501 hybrid_clock.cc:648] HybridClock initialized: now 1782009538391573 us; error 41 us; skew 500 ppm
I20260621 02:38:58.391918 7501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:58 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:58.395044 7501 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260621 02:38:58.396345 7501 webserver.cc:460] Webserver started at http://127.0.97.2:39673/ using document root <none> and password file <none>
I20260621 02:38:58.397019 7501 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:58.397171 7501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:58.402504 7501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:38:58.405155 7524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:58.406558 7501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:38:58.406713 7501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "7e54a539be224025a030c0272b000fac"
format_stamp: "Formatted at 2026-06-21 02:38:55 on dist-test-slave-jkn7"
server_key: "f6e8621464c9d963ea6123b2b6167dab"
server_key_iv: "e4772b3e6c0eaa6ba09a0db866a8915c"
server_key_version: "encryptionkey@0"
I20260621 02:38:58.407691 7501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:58.435767 7501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:58.440202 7501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:58.440514 7501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:58.441660 7501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:58.443035 7501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:58.443154 7501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:58.443280 7501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:58.443367 7501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:58.475554 7501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45449
I20260621 02:38:58.475564 7636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45449 every 8 connection(s)
I20260621 02:38:58.477008 7501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:38:58.484436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 7501
I20260621 02:38:58.484756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 6850
Jun 21 02:38:58 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 21 02:38:58 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260621 02:38:58.515820 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:58.479974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55583 (local address 127.0.97.61:40413)
0621 02:38:58.489883 (+ 9909us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:58.489889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:58.489908 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:38:58.491175 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:58.491180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:58.491250 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:58.491347 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:58.494331 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.495723 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.498119 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.498731 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.498830 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:58.504395 (+ 5565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:58.504403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:58.504408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:58.504460 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:58.508098 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:58.511103 (+ 3005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:58.511110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:58.511113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:58.511262 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:58.511780 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:58.511785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:58.511789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:58.512165 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:58.513335 (+ 1170us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:58.514670 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0621 02:38:58.514922 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9791,"thread_start_us":9700,"threads_started":1}
I20260621 02:38:58.517002 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:58.480541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.97.2:55583)
0621 02:38:58.488763 (+ 8222us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:58.488815 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:58.490722 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:58.492173 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:58.492185 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:58.493139 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:58.494092 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:58.494117 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.495921 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.495927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:58.497952 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:58.497970 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.499185 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.499192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:58.499459 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:58.500223 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:58.500253 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:58.504117 (+ 3864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:58.510223 (+ 6106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:58.510232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:58.510254 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:58.510958 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:58.511401 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:58.511405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:58.511408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:58.511647 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:58.512415 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:58.512425 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:58.514187 (+ 1762us) client_negotiation.cc:769] Sending connection context
0621 02:38:58.514519 (+ 332us) client_negotiation.cc:240] Negotiation successful
0621 02:38:58.515083 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7936,"mutex_wait_us":556,"spinlock_wait_cycles":1792,"thread_start_us":7726,"threads_started":1}
Jun 21 02:38:58 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:38:58.519027 7639 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40413
I20260621 02:38:58.519395 7639 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:58.519613 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:58.486282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.0.97.3:43879)
0621 02:38:58.490220 (+ 3938us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:58.490232 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:58.490797 (+ 565us) server_negotiation.cc:407] Connection header received
0621 02:38:58.490983 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:58.490993 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:58.491229 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:58.491374 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:58.494769 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.496846 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.499355 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.500395 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.500537 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:58.511381 (+ 10844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:58.511399 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:58.511418 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:58.511469 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:58.515044 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:58.516430 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:58.516437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:58.516440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:58.516584 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:58.517052 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:58.517057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:58.517061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:58.517494 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:58.517785 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:58.518984 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0621 02:38:58.519276 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3747,"thread_start_us":3591,"threads_started":1}
I20260621 02:38:58.520360 7639 heartbeater.cc:511] Master 127.0.97.61:40413 requested a full tablet report, sending...
Jun 21 02:38:58 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260621 02:38:58.520999 6973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:58.522487 7125 ts_manager.cc:194] Registered new tserver with Master: 7e54a539be224025a030c0272b000fac (127.0.97.2:45449)
I20260621 02:38:58.529171 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:58.479956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32999 (local address 127.0.97.62:46061)
0621 02:38:58.489552 (+ 9596us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:58.489559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:58.491832 (+ 2273us) server_negotiation.cc:407] Connection header received
0621 02:38:58.491899 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:58.491903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:58.491997 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:58.492137 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:58.503861 (+ 11724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.505238 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.512523 (+ 7285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.512958 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.513116 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:58.522611 (+ 9495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:58.522619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:58.522625 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:58.522673 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:58.526073 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:58.526775 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:58.526781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:58.526785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:58.526938 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:58.527340 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:58.527344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:58.527348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:58.527722 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:58.527871 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:58.528412 (+ 541us) server_negotiation.cc:299] Negotiation successful
0621 02:38:58.528710 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9462,"thread_start_us":9354,"threads_started":1}
I20260621 02:38:58.530642 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:58.480541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.2:32999)
0621 02:38:58.491405 (+ 10864us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:58.491423 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:58.491530 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:58.492173 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:58.492185 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:58.493139 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:58.503666 (+ 10527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:58.503684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.508292 (+ 4608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.508300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:58.509956 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:58.509970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.513149 (+ 3179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.513155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:58.513519 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:58.516958 (+ 3439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:58.516986 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:58.520856 (+ 3870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:58.526217 (+ 5361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:58.526224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:58.526228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:58.526646 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:58.527060 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:58.527066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:58.527072 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:58.527205 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:58.527946 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:58.527954 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:58.528115 (+ 161us) client_negotiation.cc:769] Sending connection context
0621 02:38:58.529398 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0621 02:38:58.529756 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10717,"mutex_wait_us":243,"spinlock_wait_cycles":24832,"thread_start_us":10461,"threads_started":1}
I20260621 02:38:58.531103 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:58.497832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45537 (local address 127.0.97.2:59433)
0621 02:38:58.508639 (+ 10807us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:58.508662 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:58.508803 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:58.509635 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:58.509643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:58.510335 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:58.510734 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:58.510747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.512458 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.512466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:58.514803 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:58.514815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.516401 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.516407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:58.516588 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:58.518856 (+ 2268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:58.518872 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:58.521581 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:58.526990 (+ 5409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:58.526996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:58.527003 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:58.527816 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:58.528501 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:58.528510 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:58.528517 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:58.528711 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:58.529534 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:58.529539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:58.529728 (+ 189us) client_negotiation.cc:769] Sending connection context
0621 02:38:58.529884 (+ 156us) client_negotiation.cc:240] Negotiation successful
0621 02:38:58.530692 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10655,"mutex_wait_us":785,"thread_start_us":10331,"threads_started":1}
I20260621 02:38:58.531458 7638 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46061
I20260621 02:38:58.531611 7638 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:58.531612 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:58.491806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59433 (local address 127.0.97.60:45537)
0621 02:38:58.507864 (+ 16058us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:58.507872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:58.508992 (+ 1120us) server_negotiation.cc:407] Connection header received
0621 02:38:58.509061 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:58.509069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:58.509158 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:58.509396 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:58.510979 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.512285 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.514973 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:58.516029 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:58.516158 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:58.521853 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:58.521863 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:58.521869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:58.521909 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:58.526811 (+ 4902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:58.528054 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:58.528061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:58.528084 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:58.528353 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:58.528886 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:58.528891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:58.528895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:58.529279 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:58.529475 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:58.530005 (+ 530us) server_negotiation.cc:299] Negotiation successful
0621 02:38:58.530645 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15876,"thread_start_us":17061,"threads_started":1}
I20260621 02:38:58.532076 7638 heartbeater.cc:511] Master 127.0.97.62:46061 requested a full tablet report, sending...
I20260621 02:38:58.532668 7637 heartbeater.cc:347] Connected to a master server at 127.0.97.60:45537
I20260621 02:38:58.532790 7637 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:58.533159 6521 ts_manager.cc:194] Re-registered known tserver with Master: 7e54a539be224025a030c0272b000fac (127.0.97.2:45449)
I20260621 02:38:58.533196 7637 heartbeater.cc:511] Master 127.0.97.60:45537 requested a full tablet report, sending...
I20260621 02:38:58.534263 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32999
I20260621 02:38:58.534369 7310 ts_manager.cc:194] Registered new tserver with Master: 7e54a539be224025a030c0272b000fac (127.0.97.2:45449)
I20260621 02:38:58.610291 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 6850
I20260621 02:38:58.626227 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43879
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46061
--webserver_port=38715
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46061,127.0.97.61:40413
--tserver_master_addrs=127.0.97.62:46061,127.0.97.61:40413,127.0.97.60:45537 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:38:58.936729 7490 heartbeater.cc:503] Master 127.0.97.62:46061 was elected leader, sending a full tablet report...
W20260621 02:38:58.953480 7650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:38:58.953804 7650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:38:58.953977 7650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:38:58.960803 7650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:38:58.960899 7650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:38:58.961078 7650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:38:58.970137 7650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38715
--tserver_master_addrs=127.0.97.62:46061,127.0.97.61:40413,127.0.97.60:45537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:38:58.971338 7650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:38:58.972626 7650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:38:58.984925 7655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:58.984951 7658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:38:58.985002 7656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:38:58.986047 7650 server_base.cc:1034] running on GCE node
I20260621 02:38:58.986613 7650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:38:58.987380 7650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:38:58.988582 7650 hybrid_clock.cc:648] HybridClock initialized: now 1782009538988536 us; error 44 us; skew 500 ppm
I20260621 02:38:58.988832 7650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:38:58 dist-test-slave-jkn7 krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:38:58.991927 7650 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260621 02:38:58.993383 7650 webserver.cc:460] Webserver started at http://127.0.97.3:38715/ using document root <none> and password file <none>
I20260621 02:38:58.993999 7650 fs_manager.cc:359] Metadata directory not provided
I20260621 02:38:58.994145 7650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:38:58.999270 7650 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:38:59.002020 7665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:59.003459 7650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:38:59.003634 7650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "26fe039cc5604f76850fe38274b4ef3f"
format_stamp: "Formatted at 2026-06-21 02:38:55 on dist-test-slave-jkn7"
server_key: "c7ef9c2adfd99887c35e0a4fd821490b"
server_key_iv: "6f5f72801956d66a04821d174eb659b5"
server_key_version: "encryptionkey@0"
I20260621 02:38:59.004665 7650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:38:59.026372 7650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:38:59.030623 7650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:38:59.030916 7650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:38:59.031869 7650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:38:59.033238 7650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:38:59.033371 7650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:59.033491 7650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:38:59.033587 7650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:38:59.065271 7650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43879
I20260621 02:38:59.065414 7777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43879 every 8 connection(s)
I20260621 02:38:59.066669 7650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:38:59.075707 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 7650
Jun 21 02:38:59 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 21 02:38:59 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260621 02:38:59.117630 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.069520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41941 (local address 127.0.97.62:46061)
0621 02:38:59.093585 (+ 24065us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.093592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.093615 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:38:59.093684 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.093690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.093790 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.093916 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.096656 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.097829 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.100647 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.101426 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.101540 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.108730 (+ 7190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.108735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.108740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.108791 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.113182 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.114288 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.114297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.114303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.114484 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.115017 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.115026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.115031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.115440 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.115602 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.116742 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.117029 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23915,"thread_start_us":23925,"threads_started":1}
I20260621 02:38:59.118700 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.089548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36243 (local address 127.0.97.60:45537)
0621 02:38:59.094089 (+ 4541us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.094097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.094119 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:38:59.094176 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.094182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.094249 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.094356 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.097931 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.099246 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.101193 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.101829 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.101944 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.109598 (+ 7654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.109605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.109610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.109649 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.113402 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.114273 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.114278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.114281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.114440 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.114944 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.114951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.114955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.115330 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.115481 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.118075 (+ 2594us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.118299 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4408,"thread_start_us":4274,"threads_started":1}
I20260621 02:38:59.119783 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.070169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.3:41941)
0621 02:38:59.089342 (+ 19173us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:59.089363 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:59.089479 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:59.094201 (+ 4722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:59.094213 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:59.095947 (+ 1734us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:59.096469 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.096495 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.097989 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.097998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.100509 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.100523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.101670 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.101681 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.102029 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.103148 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:59.103174 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:59.108505 (+ 5331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:59.113353 (+ 4848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:59.113362 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:59.113379 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:59.114123 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:59.114653 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:59.114661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:59.114668 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:59.114872 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:59.115708 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:59.115716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:59.116116 (+ 400us) client_negotiation.cc:769] Sending connection context
0621 02:38:59.116496 (+ 380us) client_negotiation.cc:240] Negotiation successful
0621 02:38:59.118328 (+ 1832us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18969,"mutex_wait_us":81,"spinlock_wait_cycles":87040,"thread_start_us":18688,"threads_started":1}
I20260621 02:38:59.120302 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.089460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45537 (local address 127.0.97.3:36243)
0621 02:38:59.089965 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:59.089979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:59.090064 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:59.094405 (+ 4341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:59.094411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:59.095308 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:59.096469 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.096495 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.099358 (+ 2863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.099363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.101030 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.101043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.101995 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.102000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.102151 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.103044 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:59.103086 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:59.109374 (+ 6288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:59.113582 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:59.113586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:59.113589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:59.114126 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:59.114562 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:59.114567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:59.114570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:59.114811 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:59.115572 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:59.115580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:59.116255 (+ 675us) client_negotiation.cc:769] Sending connection context
0621 02:38:59.116494 (+ 239us) client_negotiation.cc:240] Negotiation successful
0621 02:38:59.117069 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":166,"spinlock_wait_cycles":28032,"thread_start_us":275,"threads_started":1}
I20260621 02:38:59.121955 7778 heartbeater.cc:347] Connected to a master server at 127.0.97.60:45537
I20260621 02:38:59.122018 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.096870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36940 (local address 127.0.97.61:40413)
0621 02:38:59.097368 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.097373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.100780 (+ 3407us) server_negotiation.cc:407] Connection header received
0621 02:38:59.101013 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.101018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.101102 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.101177 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.106344 (+ 5167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.107386 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.109255 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.109988 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.110108 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.114526 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.114534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.114542 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.114576 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.118369 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.119340 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.119347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.119351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.119510 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.119957 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.119965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.119970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.120296 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.120540 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.121140 (+ 600us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.121570 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260621 02:38:59.122268 7778 heartbeater.cc:464] Registering TS with master...
Jun 21 02:38:59 dist-test-slave-jkn7 krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260621 02:38:59.123203 7778 heartbeater.cc:511] Master 127.0.97.60:45537 requested a full tablet report, sending...
I20260621 02:38:59.127877 7310 ts_manager.cc:194] Registered new tserver with Master: 26fe039cc5604f76850fe38274b4ef3f (127.0.97.3:43879)
I20260621 02:38:59.128875 7779 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46061
I20260621 02:38:59.129027 7779 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:59.129614 7779 heartbeater.cc:511] Master 127.0.97.62:46061 requested a full tablet report, sending...
I20260621 02:38:59.136710 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.070169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.97.3:42965)
0621 02:38:59.085231 (+ 15062us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:59.085318 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:59.088484 (+ 3166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:59.094667 (+ 6183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:59.094676 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:59.095399 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:38:59.096484 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.096499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.099311 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.099319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.102670 (+ 3351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.102689 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.104471 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.104481 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.104745 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.119193 (+ 14448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:38:59.119214 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:38:59.123138 (+ 3924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:38:59.132703 (+ 9565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:59.132711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:59.132718 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:59.133499 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:59.134039 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:38:59.134047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:38:59.134053 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:38:59.134253 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:38:59.135058 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:38:59.135066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:38:59.135273 (+ 207us) client_negotiation.cc:769] Sending connection context
0621 02:38:59.135543 (+ 270us) client_negotiation.cc:240] Negotiation successful
0621 02:38:59.136099 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14684,"mutex_wait_us":250,"thread_start_us":14441,"threads_started":1}
I20260621 02:38:59.138233 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.069527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42965 (local address 127.0.97.61:40413)
0621 02:38:59.094350 (+ 24823us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.094356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.094374 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:38:59.094427 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.094431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.094504 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.094624 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.098064 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.099137 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.102898 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.103782 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.103898 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.124185 (+ 20287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.124196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.124203 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.124270 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.132475 (+ 8205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.133678 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.133687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.133693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.133886 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.134398 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.134422 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.134428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.134813 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.134983 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.137217 (+ 2234us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.137547 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24691,"thread_start_us":24989,"threads_started":1}
I20260621 02:38:59.139284 6521 ts_manager.cc:194] Re-registered known tserver with Master: 26fe039cc5604f76850fe38274b4ef3f (127.0.97.3:43879)
I20260621 02:38:59.139365 7780 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40413
I20260621 02:38:59.139492 7780 heartbeater.cc:464] Registering TS with master...
I20260621 02:38:59.139922 7780 heartbeater.cc:511] Master 127.0.97.61:40413 requested a full tablet report, sending...
I20260621 02:38:59.140750 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41941
I20260621 02:38:59.141055 7125 ts_manager.cc:194] Registered new tserver with Master: 26fe039cc5604f76850fe38274b4ef3f (127.0.97.3:43879)
I20260621 02:38:59.142805 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.122869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.0.97.60:45537)
0621 02:38:59.123037 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.123045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.124717 (+ 1672us) server_negotiation.cc:407] Connection header received
0621 02:38:59.124787 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.124795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.124898 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.125028 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.127398 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.128334 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.130520 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.131179 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.131305 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.135167 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.135173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.135178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.135214 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.138767 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.139593 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.139598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.139601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.139750 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.140188 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.140193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.140196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.140534 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.140763 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.141802 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.142462 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:38:59.149248 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.097442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.0.97.62:46061)
0621 02:38:59.117928 (+ 20486us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.117936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.130272 (+ 12336us) server_negotiation.cc:407] Connection header received
0621 02:38:59.130367 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.130372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.130521 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.130608 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.133088 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.134294 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.136403 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.137483 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.137707 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.144386 (+ 6679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.144393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.144397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.144434 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.147051 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.147693 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.147696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.147699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.147805 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.148099 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.148102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.148104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.148352 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.148545 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.148778 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.148926 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20354,"thread_start_us":30167,"threads_started":1}
I20260621 02:38:59.152657 6521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:38:59.159206 6521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:38:59.193028 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.176423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.0.97.2:45449)
0621 02:38:59.185706 (+ 9283us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.185714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.185743 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:38:59.185844 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.185852 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.186135 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.186361 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:59.187251 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.188609 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.190766 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.192041 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.192181 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.192385 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.192598 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.192745 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9103,"thread_start_us":8993,"threads_started":1}
I20260621 02:38:59.193085 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.176021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45449 (local address 127.0.0.1:37658)
0621 02:38:59.185237 (+ 9216us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:59.185257 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:59.185408 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:59.186583 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:59.186587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:59.186609 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:59.187051 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.187066 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.188769 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.188774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.190627 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.190638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.192221 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.192226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.192406 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.192448 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 02:38:59.192558 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 02:38:59.192678 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9104,"mutex_wait_us":46,"thread_start_us":11784,"threads_started":1}
I20260621 02:38:59.195173 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.175596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43879 (local address 127.0.0.1:43402)
0621 02:38:59.183581 (+ 7985us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:59.183606 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:59.183755 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:59.186439 (+ 2684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:59.186447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:59.186484 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:59.186980 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.186998 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.189005 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.189010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.191770 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.191785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.194122 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.194128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.194573 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.194596 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:38:59.194709 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 02:38:59.194828 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7819,"mutex_wait_us":24,"thread_start_us":7673,"threads_started":1}
I20260621 02:38:59.195202 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.175853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.0.97.3:43879)
0621 02:38:59.185935 (+ 10082us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.185943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.185971 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:38:59.186056 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.186062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.186252 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.186415 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:59.187279 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.188877 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.191959 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.193947 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.194081 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.194299 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.194749 (+ 450us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.194891 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9940,"thread_start_us":9826,"threads_started":1}
I20260621 02:38:59.196463 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.177238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.0.97.1:38313)
0621 02:38:59.184781 (+ 7543us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.184794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.188181 (+ 3387us) server_negotiation.cc:407] Connection header received
0621 02:38:59.188328 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.188336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.188541 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.188719 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:59.189505 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.190981 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.193856 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.195195 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.195324 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.195531 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.196049 (+ 518us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.196191 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7382,"thread_start_us":7224,"threads_started":1}
I20260621 02:38:59.196992 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.176601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38313 (local address 127.0.0.1:54916)
0621 02:38:59.188053 (+ 11452us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:59.188069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:59.188178 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:59.188724 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:59.188730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:59.188758 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:59.189273 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.189311 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.191140 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.191145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.193704 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.193719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.195688 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.195694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.195850 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.195880 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:38:59.196621 (+ 741us) client_negotiation.cc:240] Negotiation successful
0621 02:38:59.196739 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11332,"thread_start_us":8382,"threads_started":1}
I20260621 02:38:59.197204 7572 tablet_service.cc:1467] Processing CreateTablet for tablet 99fbea3ce836469d99f088ec4d07acfe (DEFAULT_TABLE table=test-table [id=05e5820a291f41d2b18892f19eff0731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:59.198403 7572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99fbea3ce836469d99f088ec4d07acfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:59.198423 7713 tablet_service.cc:1467] Processing CreateTablet for tablet 99fbea3ce836469d99f088ec4d07acfe (DEFAULT_TABLE table=test-table [id=05e5820a291f41d2b18892f19eff0731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:59.199591 7713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99fbea3ce836469d99f088ec4d07acfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:59.205149 7422 tablet_service.cc:1467] Processing CreateTablet for tablet 99fbea3ce836469d99f088ec4d07acfe (DEFAULT_TABLE table=test-table [id=05e5820a291f41d2b18892f19eff0731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:38:59.206514 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99fbea3ce836469d99f088ec4d07acfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:38:59.211767 7811 tablet_bootstrap.cc:492] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: Bootstrap starting.
I20260621 02:38:59.215054 7812 tablet_bootstrap.cc:492] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: Bootstrap starting.
I20260621 02:38:59.215303 7811 tablet_bootstrap.cc:654] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:59.216572 7811 log.cc:824] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:59.218334 7812 tablet_bootstrap.cc:654] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:59.219218 7812 log.cc:824] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:59.219431 7811 tablet_bootstrap.cc:492] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: No bootstrap required, opened a new log
I20260621 02:38:59.219707 7811 ts_tablet_manager.cc:1397] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260621 02:38:59.221715 7815 tablet_bootstrap.cc:492] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: Bootstrap starting.
I20260621 02:38:59.222376 7812 tablet_bootstrap.cc:492] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: No bootstrap required, opened a new log
I20260621 02:38:59.222626 7812 ts_tablet_manager.cc:1397] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:38:59.225051 7815 tablet_bootstrap.cc:654] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: Neither blocks nor log segments found. Creating new log.
I20260621 02:38:59.226614 7815 log.cc:824] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: Log is configured to *not* fsync() on all Append() calls
I20260621 02:38:59.228247 7811 raft_consensus.cc:348] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } }
I20260621 02:38:59.228668 7811 raft_consensus.cc:374] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:59.228767 7811 raft_consensus.cc:729] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e54a539be224025a030c0272b000fac, State: Initialized, Role: FOLLOWER
I20260621 02:38:59.229414 7811 consensus_queue.cc:260] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } }
I20260621 02:38:59.229758 7815 tablet_bootstrap.cc:492] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: No bootstrap required, opened a new log
I20260621 02:38:59.230046 7815 ts_tablet_manager.cc:1397] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 02:38:59.231297 7811 ts_tablet_manager.cc:1428] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
I20260621 02:38:59.231572 7638 heartbeater.cc:503] Master 127.0.97.62:46061 was elected leader, sending a full tablet report...
I20260621 02:38:59.236797 7812 raft_consensus.cc:348] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } }
I20260621 02:38:59.237161 7812 raft_consensus.cc:374] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:59.237277 7812 raft_consensus.cc:729] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fe039cc5604f76850fe38274b4ef3f, State: Initialized, Role: FOLLOWER
I20260621 02:38:59.237821 7812 consensus_queue.cc:260] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } }
I20260621 02:38:59.238862 7779 heartbeater.cc:503] Master 127.0.97.62:46061 was elected leader, sending a full tablet report...
I20260621 02:38:59.238988 7815 raft_consensus.cc:348] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } }
I20260621 02:38:59.239354 7815 raft_consensus.cc:374] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:38:59.239441 7815 raft_consensus.cc:729] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406a3d74c2044d018ddd85d9bbff38ca, State: Initialized, Role: FOLLOWER
I20260621 02:38:59.239475 7812 ts_tablet_manager.cc:1428] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20260621 02:38:59.240031 7815 consensus_queue.cc:260] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } }
I20260621 02:38:59.242381 7815 ts_tablet_manager.cc:1428] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260621 02:38:59.319527 7781 tablet.cc:2369] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:38:59.380817 7492 tablet.cc:2369] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:38:59.480280 7640 tablet.cc:2369] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:38:59.487594 7819 raft_consensus.cc:482] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:38:59.487864 7819 raft_consensus.cc:504] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } }
I20260621 02:38:59.489584 7819 leader_election.cc:288] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26fe039cc5604f76850fe38274b4ef3f (127.0.97.3:43879), 7e54a539be224025a030c0272b000fac (127.0.97.2:45449)
I20260621 02:38:59.497153 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.489968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60671 (local address 127.0.97.3:43879)
0621 02:38:59.490190 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.490200 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.490910 (+ 710us) server_negotiation.cc:407] Connection header received
0621 02:38:59.491111 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.491116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.491174 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.491263 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:59.492264 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.492970 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.494856 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.496017 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.496141 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.496246 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.496787 (+ 541us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.496905 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:38:59.497259 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.489966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43879 (local address 127.0.97.1:60671)
0621 02:38:59.490799 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:59.490814 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:59.490979 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:59.491275 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:59.491279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:59.491300 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:59.492102 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.492110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.493103 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.493108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.494715 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.494726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.496383 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.496391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.496629 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.496662 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:38:59.496762 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:38:59.496886 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":30,"spinlock_wait_cycles":4480,"thread_start_us":486,"threads_started":1}
I20260621 02:38:59.497586 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.489956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34695 (local address 127.0.97.2:45449)
0621 02:38:59.490224 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.490233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.490889 (+ 656us) server_negotiation.cc:407] Connection header received
0621 02:38:59.491103 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.491107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.491161 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.491251 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:38:59.492195 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.492874 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.495110 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.496357 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.496486 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.496573 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.496933 (+ 360us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.497081 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:38:59.497905 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.489829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45449 (local address 127.0.97.1:34695)
0621 02:38:59.490780 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0621 02:38:59.490798 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:38:59.490969 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:38:59.491264 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:38:59.491268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:38:59.491289 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:38:59.492038 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.492047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.493007 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.493014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.494964 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:38:59.494977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.496561 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.496567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:38:59.496754 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.496787 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:38:59.497435 (+ 648us) client_negotiation.cc:240] Negotiation successful
0621 02:38:59.497551 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":54,"spinlock_wait_cycles":6912,"thread_start_us":711,"threads_started":1}
I20260621 02:38:59.498888 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fbea3ce836469d99f088ec4d07acfe" candidate_uuid: "406a3d74c2044d018ddd85d9bbff38ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e54a539be224025a030c0272b000fac" is_pre_election: true
I20260621 02:38:59.499070 7733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fbea3ce836469d99f088ec4d07acfe" candidate_uuid: "406a3d74c2044d018ddd85d9bbff38ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26fe039cc5604f76850fe38274b4ef3f" is_pre_election: true
I20260621 02:38:59.499385 7592 raft_consensus.cc:2454] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 406a3d74c2044d018ddd85d9bbff38ca in term 0.
I20260621 02:38:59.499532 7733 raft_consensus.cc:2454] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 406a3d74c2044d018ddd85d9bbff38ca in term 0.
I20260621 02:38:59.500051 7377 leader_election.cc:302] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fe039cc5604f76850fe38274b4ef3f, 406a3d74c2044d018ddd85d9bbff38ca; no voters:
I20260621 02:38:59.500619 7819 raft_consensus.cc:2781] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:38:59.500757 7819 raft_consensus.cc:482] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:38:59.500857 7819 raft_consensus.cc:3037] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:59.503840 7819 raft_consensus.cc:504] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } }
I20260621 02:38:59.504812 7819 leader_election.cc:288] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [CANDIDATE]: Term 1 election: Requested vote from peers 26fe039cc5604f76850fe38274b4ef3f (127.0.97.3:43879), 7e54a539be224025a030c0272b000fac (127.0.97.2:45449)
I20260621 02:38:59.504837 7733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fbea3ce836469d99f088ec4d07acfe" candidate_uuid: "406a3d74c2044d018ddd85d9bbff38ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26fe039cc5604f76850fe38274b4ef3f"
I20260621 02:38:59.504909 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fbea3ce836469d99f088ec4d07acfe" candidate_uuid: "406a3d74c2044d018ddd85d9bbff38ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e54a539be224025a030c0272b000fac"
I20260621 02:38:59.505023 7733 raft_consensus.cc:3037] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:59.505084 7592 raft_consensus.cc:3037] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:38:59.507557 7592 raft_consensus.cc:2454] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 406a3d74c2044d018ddd85d9bbff38ca in term 1.
I20260621 02:38:59.508077 7375 leader_election.cc:302] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 406a3d74c2044d018ddd85d9bbff38ca, 7e54a539be224025a030c0272b000fac; no voters:
I20260621 02:38:59.508136 7733 raft_consensus.cc:2454] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 406a3d74c2044d018ddd85d9bbff38ca in term 1.
I20260621 02:38:59.508374 7819 raft_consensus.cc:2781] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:38:59.508844 7819 raft_consensus.cc:686] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 1 LEADER]: Becoming Leader. State: Replica: 406a3d74c2044d018ddd85d9bbff38ca, State: Running, Role: LEADER
I20260621 02:38:59.509410 7819 consensus_queue.cc:237] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } }
I20260621 02:38:59.515048 6522 catalog_manager.cc:5495] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 406a3d74c2044d018ddd85d9bbff38ca (127.0.97.1). New cstate: current_term: 1 leader_uuid: "406a3d74c2044d018ddd85d9bbff38ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } health_report { overall_health: HEALTHY } } }
I20260621 02:38:59.554409 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.549398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.0.97.1:38313)
0621 02:38:59.549578 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.549583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.549671 (+ 88us) server_negotiation.cc:407] Connection header received
0621 02:38:59.549853 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.549856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.549936 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.550063 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:38:59.550390 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.551092 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.552373 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.552802 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.552906 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.553347 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:38:59.553880 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:38:59.554002 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.554143 (+ 141us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.554209 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:38:59.562691 7592 raft_consensus.cc:1261] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [term 1 FOLLOWER]: Refusing update from remote peer 406a3d74c2044d018ddd85d9bbff38ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:59.562691 7733 raft_consensus.cc:1261] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [term 1 FOLLOWER]: Refusing update from remote peer 406a3d74c2044d018ddd85d9bbff38ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:38:59.563637 7822 consensus_queue.cc:1035] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:59.563997 7819 consensus_queue.cc:1035] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:38:59.577845 7825 mvcc.cc:205] Tried to move back new op lower bound from 7299111074038349824 to 7299111073834201088. Current Snapshot: MvccSnapshot[applied={T|T < 7299111074038349824}]
I20260621 02:38:59.579429 7826 mvcc.cc:205] Tried to move back new op lower bound from 7299111074038349824 to 7299111073834201088. Current Snapshot: MvccSnapshot[applied={T|T < 7299111074038349824}]
I20260621 02:38:59.597203 6521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:59.597548 6521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:38:59.605618 6521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1522f392977c491790c78c87c51a46b6: Sending DeleteTablet for 3 replicas of tablet 99fbea3ce836469d99f088ec4d07acfe
I20260621 02:38:59.606851 7713 tablet_service.cc:1514] Processing DeleteTablet for tablet 99fbea3ce836469d99f088ec4d07acfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43402
I20260621 02:38:59.607010 7572 tablet_service.cc:1514] Processing DeleteTablet for tablet 99fbea3ce836469d99f088ec4d07acfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37658
I20260621 02:38:59.607105 7422 tablet_service.cc:1514] Processing DeleteTablet for tablet 99fbea3ce836469d99f088ec4d07acfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:38:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54916
I20260621 02:38:59.608000 7834 tablet_replica.cc:330] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: stopping tablet replica
I20260621 02:38:59.608563 7834 raft_consensus.cc:2229] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:59.609150 7834 raft_consensus.cc:2258] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:59.613039 7834 ts_tablet_manager.cc:1905] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:59.620712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 6490
I20260621 02:38:59.621755 7834 ts_tablet_manager.cc:1918] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:59.621954 7834 log.cc:1196] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/99fbea3ce836469d99f088ec4d07acfe
I20260621 02:38:59.621984 6552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:38:59.623221 7834 ts_tablet_manager.cc:1939] T 99fbea3ce836469d99f088ec4d07acfe P 26fe039cc5604f76850fe38274b4ef3f: Deleting consensus metadata
I20260621 02:38:59.634425 7835 tablet_replica.cc:330] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: stopping tablet replica
I20260621 02:38:59.635092 7836 tablet_replica.cc:330] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: stopping tablet replica
I20260621 02:38:59.635689 7836 raft_consensus.cc:2229] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:38:59.636299 7835 raft_consensus.cc:2229] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:38:59.638067 7836 raft_consensus.cc:2258] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:59.638615 7835 raft_consensus.cc:2258] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:38:59.642377 7835 ts_tablet_manager.cc:1905] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:59.643692 7836 ts_tablet_manager.cc:1905] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:38:59.649328 7835 ts_tablet_manager.cc:1918] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:59.649485 7835 log.cc:1196] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/99fbea3ce836469d99f088ec4d07acfe
I20260621 02:38:59.650028 7835 ts_tablet_manager.cc:1939] T 99fbea3ce836469d99f088ec4d07acfe P 7e54a539be224025a030c0272b000fac: Deleting consensus metadata
I20260621 02:38:59.655607 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.610261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.0.97.61:40413)
0621 02:38:59.610427 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.610433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.619140 (+ 8707us) server_negotiation.cc:407] Connection header received
0621 02:38:59.628098 (+ 8958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.628104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.628223 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.628285 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.630769 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.631946 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.636552 (+ 4606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.637203 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.637333 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.647832 (+ 10499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.647841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.647848 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.647922 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.651805 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.652634 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.652640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.652644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.652784 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.653233 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.653238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.653241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.653631 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.653848 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.655025 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.655276 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:38:59.661199 7836 ts_tablet_manager.cc:1918] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:38:59.661460 7836 log.cc:1196] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/99fbea3ce836469d99f088ec4d07acfe
I20260621 02:38:59.662133 7836 ts_tablet_manager.cc:1939] T 99fbea3ce836469d99f088ec4d07acfe P 406a3d74c2044d018ddd85d9bbff38ca: Deleting consensus metadata
I20260621 02:38:59.671783 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.619692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.0.97.60:45537)
0621 02:38:59.619879 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.619885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.656610 (+ 36725us) server_negotiation.cc:407] Connection header received
0621 02:38:59.656846 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.656853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.656931 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.656993 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.658955 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.660057 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.662134 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.662736 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.662834 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.666096 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.666101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.666104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.666138 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.669168 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.669845 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.669855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.669857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.669973 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.670335 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.670339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.670343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.670622 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.670895 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.671188 (+ 293us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.671398 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:38:59.793077 6507 catalog_manager.cc:4841] TS 406a3d74c2044d018ddd85d9bbff38ca (127.0.97.1:38313): tablet 99fbea3ce836469d99f088ec4d07acfe (table test-table [id=05e5820a291f41d2b18892f19eff0731]) successfully deleted
I20260621 02:38:59.793047 6506 catalog_manager.cc:4841] TS 7e54a539be224025a030c0272b000fac (127.0.97.2:45449): tablet 99fbea3ce836469d99f088ec4d07acfe (table test-table [id=05e5820a291f41d2b18892f19eff0731]) successfully deleted
I20260621 02:38:59.793740 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 6490
I20260621 02:38:59.836526 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.820379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.0.97.60:45537)
0621 02:38:59.820590 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.820595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.821471 (+ 876us) server_negotiation.cc:407] Connection header received
0621 02:38:59.821675 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.821681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.821759 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.821857 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.823872 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.824551 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.826722 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.827355 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.827482 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.830824 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.830831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.830836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.830871 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.833844 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.834448 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.834453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.834457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.834576 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.834911 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.834915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.834919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.835245 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.835491 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.836006 (+ 515us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.836197 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:38:59.838409 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:38:59.819404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.0.97.61:40413)
0621 02:38:59.819644 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 02:38:59.819654 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:38:59.820513 (+ 859us) server_negotiation.cc:407] Connection header received
0621 02:38:59.820798 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:38:59.820804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:38:59.820883 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:38:59.820989 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:38:59.823383 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.824502 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.827661 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:38:59.828285 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:38:59.828408 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:38:59.831609 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:38:59.831618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:38:59.831622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:38:59.831658 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:38:59.835043 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.835765 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.835771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.835774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.835909 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:38:59.837005 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:38:59.837011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:38:59.837014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:38:59.837266 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:38:59.837598 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0621 02:38:59.837896 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 02:38:59.838113 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:39:00.585470 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:00.584661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.1:45653)
0621 02:39:00.585111 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:00.585242 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":207,"threads_started":1}
W20260621 02:39:00.585882 7490 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46061 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
I20260621 02:39:00.590365 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:00.589517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.3:35377)
0621 02:39:00.589978 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:00.590156 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":238,"threads_started":1}
W20260621 02:39:00.590729 7779 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46061 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
I20260621 02:39:00.594630 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:00.594029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.2:45503)
0621 02:39:00.594382 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:00.594482 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":172,"threads_started":1}
W20260621 02:39:00.595029 7638 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46061 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
I20260621 02:39:01.106060 7849 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1522f392977c491790c78c87c51a46b6)
I20260621 02:39:01.106377 7849 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } }
I20260621 02:39:01.106683 7848 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1522f392977c491790c78c87c51a46b6)
I20260621 02:39:01.106971 7848 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } }
I20260621 02:39:01.109092 7848 leader_election.cc:288] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1522f392977c491790c78c87c51a46b6 (127.0.97.62:46061), c5dcd042c98242c89b56db2487519736 (127.0.97.60:45537)
I20260621 02:39:01.113715 7849 leader_election.cc:288] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1522f392977c491790c78c87c51a46b6 (127.0.97.62:46061), d8b7d45d78264253924c1d843c95af9a (127.0.97.61:40413)
I20260621 02:39:01.120813 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.107753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60634)
0621 02:39:01.120277 (+ 12524us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.120456 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12341,"threads_started":1}
I20260621 02:39:01.121165 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.109485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60644)
0621 02:39:01.120599 (+ 11114us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.120809 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10974,"thread_start_us":10798,"threads_started":1}
W20260621 02:39:01.121850 7112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
W20260621 02:39:01.124577 7297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
I20260621 02:39:01.125149 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.123875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60662)
0621 02:39:01.124518 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.124726 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":333,"threads_started":1}
W20260621 02:39:01.125422 7112 leader_election.cc:334] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1522f392977c491790c78c87c51a46b6 (127.0.97.62:46061): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
I20260621 02:39:01.127336 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.126718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60668)
0621 02:39:01.126917 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.127090 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260621 02:39:01.127616 7297 leader_election.cc:334] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1522f392977c491790c78c87c51a46b6 (127.0.97.62:46061): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
I20260621 02:39:01.129900 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.107791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40413 (local address 127.0.0.1:36974)
0621 02:39:01.121175 (+ 13384us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.121200 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:01.121536 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:01.121992 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:01.121997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:01.122030 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:01.122328 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.122342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.125139 (+ 2797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.125147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.127425 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.127452 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.129222 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.129235 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.129431 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.129456 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 02:39:01.129539 (+ 83us) client_negotiation.cc:240] Negotiation successful
0621 02:39:01.129640 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13270,"thread_start_us":12115,"threads_started":1}
I20260621 02:39:01.130154 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.107755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.0.97.61:40413)
0621 02:39:01.119970 (+ 12215us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.119977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.121304 (+ 1327us) server_negotiation.cc:407] Connection header received
0621 02:39:01.121711 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.121716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.121790 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.122492 (+ 702us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:01.122825 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.124049 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.127615 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.129044 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.129168 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.129274 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.129580 (+ 306us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.129721 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":12072,"thread_start_us":11839,"threads_started":1}
I20260621 02:39:01.130692 7135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5dcd042c98242c89b56db2487519736" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d8b7d45d78264253924c1d843c95af9a" is_pre_election: true
I20260621 02:39:01.131001 7135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5dcd042c98242c89b56db2487519736 in term 1.
I20260621 02:39:01.131472 7294 leader_election.cc:302] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5dcd042c98242c89b56db2487519736, d8b7d45d78264253924c1d843c95af9a; no voters: 1522f392977c491790c78c87c51a46b6
I20260621 02:39:01.131865 7849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 02:39:01.132001 7849 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1522f392977c491790c78c87c51a46b6)
I20260621 02:39:01.132074 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.121542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45537 (local address 127.0.0.1:55244)
0621 02:39:01.122444 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.122462 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:01.122588 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:01.125670 (+ 3082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:01.125678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:01.125714 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:01.125952 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.125962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.127745 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.127750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.129731 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.129743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.131357 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.131363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.131534 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.131556 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:39:01.131667 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 02:39:01.131805 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776}
I20260621 02:39:01.132134 7849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:39:01.132282 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.120968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.0.97.60:45537)
0621 02:39:01.125221 (+ 4253us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.125229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.125253 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:39:01.125374 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.125382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.125479 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.126102 (+ 623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:01.126443 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.127528 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.129904 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.131128 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.131325 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.131427 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.131704 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.131825 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":4085,"thread_start_us":3940,"threads_started":1}
I20260621 02:39:01.132817 7320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8b7d45d78264253924c1d843c95af9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c5dcd042c98242c89b56db2487519736" is_pre_election: true
I20260621 02:39:01.134536 7849 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } }
I20260621 02:39:01.134960 7320 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8b7d45d78264253924c1d843c95af9a in current term 2: Already voted for candidate c5dcd042c98242c89b56db2487519736 in this term.
I20260621 02:39:01.135185 7849 leader_election.cc:288] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [CANDIDATE]: Term 2 election: Requested vote from peers 1522f392977c491790c78c87c51a46b6 (127.0.97.62:46061), d8b7d45d78264253924c1d843c95af9a (127.0.97.61:40413)
I20260621 02:39:01.135620 7135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5dcd042c98242c89b56db2487519736" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d8b7d45d78264253924c1d843c95af9a"
I20260621 02:39:01.135740 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.135285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60676)
0621 02:39:01.135492 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.135571 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260621 02:39:01.135847 7135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:39:01.136003 7112 leader_election.cc:302] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8b7d45d78264253924c1d843c95af9a; no voters: 1522f392977c491790c78c87c51a46b6, c5dcd042c98242c89b56db2487519736
I20260621 02:39:01.136665 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.136351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60688)
0621 02:39:01.136493 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.136569 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:39:01.136869 7297 leader_election.cc:334] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1522f392977c491790c78c87c51a46b6 (127.0.97.62:46061): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
I20260621 02:39:01.138209 7135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5dcd042c98242c89b56db2487519736 in term 2.
I20260621 02:39:01.138366 7848 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:39:01.138557 7294 leader_election.cc:302] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5dcd042c98242c89b56db2487519736, d8b7d45d78264253924c1d843c95af9a; no voters: 1522f392977c491790c78c87c51a46b6
I20260621 02:39:01.138821 7849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:39:01.139192 7849 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [term 2 LEADER]: Becoming Leader. State: Replica: c5dcd042c98242c89b56db2487519736, State: Running, Role: LEADER
I20260621 02:39:01.139534 7849 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } }
I20260621 02:39:01.141382 7858 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5dcd042c98242c89b56db2487519736. Latest consensus state: current_term: 2 leader_uuid: "c5dcd042c98242c89b56db2487519736" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:39:01.141779 7858 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:01.142408 7860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:39:01.145756 7860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:39:01.146528 7860 catalog_manager.cc:1259] Loaded cluster ID: 0c06c1c544434673a79cfe1a0e53415f
I20260621 02:39:01.146646 7860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:39:01.147838 7860 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:39:01.148654 7860 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Loaded TSK: 0
I20260621 02:39:01.149509 7860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:39:01.180106 7309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:39:01.187362 7309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:39:01.191195 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.190792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60708)
0621 02:39:01.190978 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.191059 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:39:01.191220 7135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [term 2 FOLLOWER]: Refusing update from remote peer c5dcd042c98242c89b56db2487519736: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260621 02:39:01.191977 7858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260621 02:39:01.192142 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.191777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:60724)
0621 02:39:01.191930 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.192028 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 02:39:01.192278 7297 consensus_peers.cc:487] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 -> Peer 1522f392977c491790c78c87c51a46b6 (127.0.97.62:46061): Couldn't send request to peer 1522f392977c491790c78c87c51a46b6. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 02:39:01.194325 7848 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5dcd042c98242c89b56db2487519736. Latest consensus state: current_term: 2 leader_uuid: "c5dcd042c98242c89b56db2487519736" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:39:01.194595 7848 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:39:01.196027 7858 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c5dcd042c98242c89b56db2487519736" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:39:01.196465 7858 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:01.196667 7848 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c5dcd042c98242c89b56db2487519736" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1522f392977c491790c78c87c51a46b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46061 } } peers { permanent_uuid: "d8b7d45d78264253924c1d843c95af9a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40413 } attrs { promote: false } } peers { permanent_uuid: "c5dcd042c98242c89b56db2487519736" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45537 } attrs { promote: false } } }
I20260621 02:39:01.196908 7848 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b7d45d78264253924c1d843c95af9a [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:39:01.211676 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.204342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45449 (local address 127.0.0.1:37674)
0621 02:39:01.204493 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.204512 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:01.204641 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:01.205536 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:01.205543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:01.205572 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:01.205761 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.205771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.207167 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.207172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.209211 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.209224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.210892 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.210898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.211079 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.211114 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:39:01.211178 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 02:39:01.211315 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 02:39:01.212900 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.204687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.0.97.2:45449)
0621 02:39:01.205176 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.205182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.205201 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:39:01.205334 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.205339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.205414 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.205525 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:01.205931 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.206973 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.209508 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.210717 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.212292 (+ 1575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.212404 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.212452 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.212601 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
I20260621 02:39:01.213608 7572 tablet_service.cc:1467] Processing CreateTablet for tablet 37e671fb931649f3b282cd1ed6708309 (DEFAULT_TABLE table=test-table [id=5ae104e54950416f82308560b1fad602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:01.214051 7572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e671fb931649f3b282cd1ed6708309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:01.219470 7871 tablet_bootstrap.cc:492] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: Bootstrap starting.
I20260621 02:39:01.221374 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.204765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38313 (local address 127.0.0.1:54930)
0621 02:39:01.205158 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.205175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:01.205273 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:01.205746 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:01.205749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:01.205773 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:01.206199 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.206216 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.208754 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.208760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.210913 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.210923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.220487 (+ 9564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.220493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.220700 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.220746 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 02:39:01.220823 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 02:39:01.220993 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278}
I20260621 02:39:01.223001 7871 tablet_bootstrap.cc:654] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:01.223979 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.204890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.0.97.1:38313)
0621 02:39:01.205316 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.205322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.205343 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:39:01.205414 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.205419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.205495 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.205878 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:01.206372 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.208594 (+ 2222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.212356 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.220230 (+ 7874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.223327 (+ 3097us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.223441 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.223500 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.223660 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":489,"threads_started":1}
I20260621 02:39:01.224913 7422 tablet_service.cc:1467] Processing CreateTablet for tablet 37e671fb931649f3b282cd1ed6708309 (DEFAULT_TABLE table=test-table [id=5ae104e54950416f82308560b1fad602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:01.225485 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e671fb931649f3b282cd1ed6708309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:01.226220 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.205708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43879 (local address 127.0.0.1:43414)
0621 02:39:01.211914 (+ 6206us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.211934 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:01.212029 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:01.218591 (+ 6562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:01.218596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:01.218628 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:01.218821 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.218832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.220750 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.220759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.223093 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.223106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.225149 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.225157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.225496 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.225551 (+ 55us) client_negotiation.cc:769] Sending connection context
0621 02:39:01.225642 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 02:39:01.225821 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6119,"threads_started":1}
I20260621 02:39:01.227259 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.205823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.0.97.3:43879)
0621 02:39:01.218310 (+ 12487us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.218316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.218332 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:39:01.218389 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.218392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.218449 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.218533 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:01.218994 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.220505 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.223256 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.224911 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.226452 (+ 1541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.226610 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.226674 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.226862 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12374,"thread_start_us":12273,"threads_started":1}
I20260621 02:39:01.231168 7871 tablet_bootstrap.cc:492] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: No bootstrap required, opened a new log
I20260621 02:39:01.231319 7871 ts_tablet_manager.cc:1397] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260621 02:39:01.232116 7871 raft_consensus.cc:348] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } }
I20260621 02:39:01.232374 7871 raft_consensus.cc:374] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:01.232483 7871 raft_consensus.cc:729] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e54a539be224025a030c0272b000fac, State: Initialized, Role: FOLLOWER
I20260621 02:39:01.232762 7871 consensus_queue.cc:260] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } }
I20260621 02:39:01.233664 7713 tablet_service.cc:1467] Processing CreateTablet for tablet 37e671fb931649f3b282cd1ed6708309 (DEFAULT_TABLE table=test-table [id=5ae104e54950416f82308560b1fad602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:01.233754 7871 ts_tablet_manager.cc:1428] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 02:39:01.234072 7713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e671fb931649f3b282cd1ed6708309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:01.236112 7873 tablet_bootstrap.cc:492] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: Bootstrap starting.
I20260621 02:39:01.236351 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.235418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.2:32815)
0621 02:39:01.235974 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.236140 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"threads_started":1}
I20260621 02:39:01.237850 7873 tablet_bootstrap.cc:654] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:01.238466 7637 heartbeater.cc:503] Master 127.0.97.60:45537 was elected leader, sending a full tablet report...
I20260621 02:39:01.240645 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.239098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.2:53371)
0621 02:39:01.240409 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.240507 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1205}
I20260621 02:39:01.240674 7873 tablet_bootstrap.cc:492] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: No bootstrap required, opened a new log
I20260621 02:39:01.240804 7873 ts_tablet_manager.cc:1397] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 02:39:01.241460 7877 tablet_bootstrap.cc:492] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: Bootstrap starting.
I20260621 02:39:01.241629 7873 raft_consensus.cc:348] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } }
I20260621 02:39:01.241851 7873 raft_consensus.cc:374] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:01.241946 7873 raft_consensus.cc:729] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406a3d74c2044d018ddd85d9bbff38ca, State: Initialized, Role: FOLLOWER
I20260621 02:39:01.242178 7873 consensus_queue.cc:260] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } }
I20260621 02:39:01.242972 7873 ts_tablet_manager.cc:1428] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 02:39:01.243880 7877 tablet_bootstrap.cc:654] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:01.244078 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.243186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.1:44087)
0621 02:39:01.243735 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.243842 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
I20260621 02:39:01.245103 7489 heartbeater.cc:503] Master 127.0.97.60:45537 was elected leader, sending a full tablet report...
I20260621 02:39:01.246019 7877 tablet_bootstrap.cc:492] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: No bootstrap required, opened a new log
I20260621 02:39:01.246157 7877 ts_tablet_manager.cc:1397] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260621 02:39:01.246699 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.245700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.1:47877)
0621 02:39:01.246501 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.246580 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 02:39:01.246907 7877 raft_consensus.cc:348] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } }
I20260621 02:39:01.247148 7877 raft_consensus.cc:374] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:01.247241 7877 raft_consensus.cc:729] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fe039cc5604f76850fe38274b4ef3f, State: Initialized, Role: FOLLOWER
I20260621 02:39:01.247473 7877 consensus_queue.cc:260] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } }
I20260621 02:39:01.248210 7877 ts_tablet_manager.cc:1428] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:39:01.249416 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.248515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.3:41483)
0621 02:39:01.249028 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.249143 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"threads_started":1}
I20260621 02:39:01.250406 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.250027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.3:42529)
0621 02:39:01.250181 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.250276 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 02:39:01.250926 7778 heartbeater.cc:503] Master 127.0.97.60:45537 was elected leader, sending a full tablet report...
I20260621 02:39:01.324052 7881 raft_consensus.cc:482] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:39:01.324298 7881 raft_consensus.cc:504] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } }
I20260621 02:39:01.326602 7881 leader_election.cc:288] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e54a539be224025a030c0272b000fac (127.0.97.2:45449), 406a3d74c2044d018ddd85d9bbff38ca (127.0.97.1:38313)
I20260621 02:39:01.331790 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.326272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37667 (local address 127.0.97.2:45449)
0621 02:39:01.326482 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.326487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.326580 (+ 93us) server_negotiation.cc:407] Connection header received
0621 02:39:01.326994 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.326998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.327060 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.327157 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:01.327574 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.328242 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.330061 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.330910 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.331055 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.331160 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.331409 (+ 249us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.331529 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:39:01.331895 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.326166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45449 (local address 127.0.97.3:37667)
0621 02:39:01.326441 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.326464 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:01.326719 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:01.327177 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:01.327181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:01.327227 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:01.327438 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.327446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.328373 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.328378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.329912 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.329924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.331089 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.331095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.331260 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.331283 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:39:01.331386 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:39:01.331528 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":18}
I20260621 02:39:01.331941 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.326281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34337 (local address 127.0.97.1:38313)
0621 02:39:01.326477 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.326483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.327051 (+ 568us) server_negotiation.cc:407] Connection header received
0621 02:39:01.327240 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.327245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.327309 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.327401 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:01.327887 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.328536 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.330260 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.331080 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.331223 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.331321 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.331569 (+ 248us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.331685 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:39:01.332091 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.326233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38313 (local address 127.0.97.3:34337)
0621 02:39:01.326945 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.326958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:01.327103 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:01.327425 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:01.327430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:01.327453 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:01.327746 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.327754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.328663 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.328671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.330128 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:01.330139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.331257 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.331263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:01.331418 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.331441 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:39:01.331546 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 02:39:01.331668 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":19,"thread_start_us":455,"threads_started":1}
I20260621 02:39:01.332494 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e671fb931649f3b282cd1ed6708309" candidate_uuid: "26fe039cc5604f76850fe38274b4ef3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e54a539be224025a030c0272b000fac" is_pre_election: true
I20260621 02:39:01.332763 7592 raft_consensus.cc:2454] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26fe039cc5604f76850fe38274b4ef3f in term 0.
I20260621 02:39:01.333242 7666 leader_election.cc:302] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fe039cc5604f76850fe38274b4ef3f, 7e54a539be224025a030c0272b000fac; no voters:
I20260621 02:39:01.333259 7442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e671fb931649f3b282cd1ed6708309" candidate_uuid: "26fe039cc5604f76850fe38274b4ef3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406a3d74c2044d018ddd85d9bbff38ca" is_pre_election: true
I20260621 02:39:01.333611 7881 raft_consensus.cc:2781] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:39:01.333731 7881 raft_consensus.cc:482] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:39:01.333750 7442 raft_consensus.cc:2454] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26fe039cc5604f76850fe38274b4ef3f in term 0.
I20260621 02:39:01.333832 7881 raft_consensus.cc:3037] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:01.335992 7881 raft_consensus.cc:504] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } }
I20260621 02:39:01.336542 7881 leader_election.cc:288] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [CANDIDATE]: Term 1 election: Requested vote from peers 7e54a539be224025a030c0272b000fac (127.0.97.2:45449), 406a3d74c2044d018ddd85d9bbff38ca (127.0.97.1:38313)
I20260621 02:39:01.336979 7442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e671fb931649f3b282cd1ed6708309" candidate_uuid: "26fe039cc5604f76850fe38274b4ef3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406a3d74c2044d018ddd85d9bbff38ca"
I20260621 02:39:01.336994 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e671fb931649f3b282cd1ed6708309" candidate_uuid: "26fe039cc5604f76850fe38274b4ef3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e54a539be224025a030c0272b000fac"
I20260621 02:39:01.337152 7442 raft_consensus.cc:3037] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:01.337179 7592 raft_consensus.cc:3037] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:01.339090 7592 raft_consensus.cc:2454] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26fe039cc5604f76850fe38274b4ef3f in term 1.
I20260621 02:39:01.339148 7442 raft_consensus.cc:2454] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26fe039cc5604f76850fe38274b4ef3f in term 1.
I20260621 02:39:01.339542 7667 leader_election.cc:302] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26fe039cc5604f76850fe38274b4ef3f, 406a3d74c2044d018ddd85d9bbff38ca, 7e54a539be224025a030c0272b000fac; no voters:
I20260621 02:39:01.339871 7881 raft_consensus.cc:2781] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:01.340356 7881 raft_consensus.cc:686] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 1 LEADER]: Becoming Leader. State: Replica: 26fe039cc5604f76850fe38274b4ef3f, State: Running, Role: LEADER
I20260621 02:39:01.341055 7881 consensus_queue.cc:237] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } }
I20260621 02:39:01.341463 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.340923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.3:37433)
0621 02:39:01.341106 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.341212 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:39:01.344875 7309 catalog_manager.cc:5495] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 26fe039cc5604f76850fe38274b4ef3f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "26fe039cc5604f76850fe38274b4ef3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26fe039cc5604f76850fe38274b4ef3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43879 } health_report { overall_health: HEALTHY } } }
I20260621 02:39:01.367684 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.362752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.0.97.3:43879)
0621 02:39:01.362914 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.362919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.362936 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:39:01.363080 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.363084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.363154 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.363271 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:01.363544 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.364205 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.365624 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.366038 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.366140 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.366553 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:01.367076 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:01.367193 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.367296 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.367363 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 02:39:01.372365 7442 raft_consensus.cc:1261] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [term 1 FOLLOWER]: Refusing update from remote peer 26fe039cc5604f76850fe38274b4ef3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:01.373219 7592 raft_consensus.cc:1261] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [term 1 FOLLOWER]: Refusing update from remote peer 26fe039cc5604f76850fe38274b4ef3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:01.373342 7884 consensus_queue.cc:1035] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "406a3d74c2044d018ddd85d9bbff38ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:01.373874 7881 consensus_queue.cc:1035] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e54a539be224025a030c0272b000fac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:01.376049 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.375542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.2:46311)
0621 02:39:01.375761 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.375847 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260621 02:39:01.376834 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.376383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.1:46673)
0621 02:39:01.376525 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.376642 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260621 02:39:01.382007 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.381419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.2:59045)
0621 02:39:01.381692 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.381794 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260621 02:39:01.383467 7888 mvcc.cc:205] Tried to move back new op lower bound from 7299111081452384256 to 7299111081337298944. Current Snapshot: MvccSnapshot[applied={T|T < 7299111081452384256}]
I20260621 02:39:01.393555 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.392788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.3:44187)
0621 02:39:01.393237 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.393356 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328}
I20260621 02:39:01.394047 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.393503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.1:44153)
0621 02:39:01.393731 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.393848 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260621 02:39:01.397171 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.396103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.97.3:41237)
0621 02:39:01.396955 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.397039 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":757}
I20260621 02:39:01.405267 7309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:01.405596 7309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:01.410959 7309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5dcd042c98242c89b56db2487519736: Sending DeleteTablet for 3 replicas of tablet 37e671fb931649f3b282cd1ed6708309
I20260621 02:39:01.412225 7713 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e671fb931649f3b282cd1ed6708309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:01 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:43414
I20260621 02:39:01.412252 7572 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e671fb931649f3b282cd1ed6708309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:01 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:37674
I20260621 02:39:01.412367 7422 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e671fb931649f3b282cd1ed6708309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:01 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:54930
I20260621 02:39:01.414342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 7282
I20260621 02:39:01.422796 7896 tablet_replica.cc:330] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: stopping tablet replica
I20260621 02:39:01.423020 7896 raft_consensus.cc:2229] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:39:01.423671 7896 raft_consensus.cc:2258] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:01.424325 7897 tablet_replica.cc:330] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: stopping tablet replica
I20260621 02:39:01.424562 7897 raft_consensus.cc:2229] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:01.424880 7897 raft_consensus.cc:2258] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:01.426328 7898 tablet_replica.cc:330] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: stopping tablet replica
I20260621 02:39:01.426566 7898 raft_consensus.cc:2229] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:01.426906 7898 raft_consensus.cc:2258] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:01.428023 7898 ts_tablet_manager.cc:1905] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:01.430277 7897 ts_tablet_manager.cc:1905] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:01.430923 7896 ts_tablet_manager.cc:1905] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:01.435011 7898 ts_tablet_manager.cc:1918] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:01.435190 7898 log.cc:1196] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/37e671fb931649f3b282cd1ed6708309
I20260621 02:39:01.435722 7898 ts_tablet_manager.cc:1939] T 37e671fb931649f3b282cd1ed6708309 P 406a3d74c2044d018ddd85d9bbff38ca: Deleting consensus metadata
I20260621 02:39:01.437019 7295 catalog_manager.cc:4841] TS 406a3d74c2044d018ddd85d9bbff38ca (127.0.97.1:38313): tablet 37e671fb931649f3b282cd1ed6708309 (table test-table [id=5ae104e54950416f82308560b1fad602]) successfully deleted
I20260621 02:39:01.441670 7896 ts_tablet_manager.cc:1918] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:01.442243 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.415902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.0.97.1:38313)
0621 02:39:01.416098 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.416104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.425719 (+ 9615us) server_negotiation.cc:407] Connection header received
0621 02:39:01.425923 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.425935 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.426064 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.426185 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:01.428164 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.429225 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.431499 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.432127 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.432230 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.435801 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:01.435817 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:01.435838 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:01.435878 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:01.439299 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:01.439979 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:01.439987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:01.439990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:01.440114 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:01.440472 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:01.440479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:01.440483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:01.440947 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:01.441258 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.441626 (+ 368us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.441837 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:39:01.442941 7482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:01.445274 7897 ts_tablet_manager.cc:1918] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:01.446934 7897 log.cc:1196] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/37e671fb931649f3b282cd1ed6708309
I20260621 02:39:01.447413 7897 ts_tablet_manager.cc:1939] T 37e671fb931649f3b282cd1ed6708309 P 7e54a539be224025a030c0272b000fac: Deleting consensus metadata
I20260621 02:39:01.448549 7896 log.cc:1196] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/37e671fb931649f3b282cd1ed6708309
I20260621 02:39:01.448693 7294 catalog_manager.cc:4841] TS 7e54a539be224025a030c0272b000fac (127.0.97.2:45449): tablet 37e671fb931649f3b282cd1ed6708309 (table test-table [id=5ae104e54950416f82308560b1fad602]) successfully deleted
I20260621 02:39:01.449081 7896 ts_tablet_manager.cc:1939] T 37e671fb931649f3b282cd1ed6708309 P 26fe039cc5604f76850fe38274b4ef3f: Deleting consensus metadata
I20260621 02:39:01.451797 7296 catalog_manager.cc:4841] TS 26fe039cc5604f76850fe38274b4ef3f (127.0.97.3:43879): tablet 37e671fb931649f3b282cd1ed6708309 (table test-table [id=5ae104e54950416f82308560b1fad602]) successfully deleted
I20260621 02:39:01.576285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 7282
I20260621 02:39:01.599740 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 7501
I20260621 02:39:01.614892 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.600581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.0.97.2:45449)
0621 02:39:01.600767 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.600773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.600871 (+ 98us) server_negotiation.cc:407] Connection header received
0621 02:39:01.601087 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.601097 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.601159 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.601261 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:01.602634 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.603380 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.604565 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.605041 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.605162 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.608640 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:01.608655 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:01.608671 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:01.608724 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:01.611914 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:01.612785 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:01.612792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:01.612795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:01.612921 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:01.613395 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:01.613404 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:01.613410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:01.613865 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:01.614285 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.614478 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.614641 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:39:01.615608 7632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:01.668361 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.667810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:54654)
0621 02:39:01.668026 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.668149 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 02:39:01.669512 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.669116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:54666)
0621 02:39:01.669266 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:01.669370 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 02:39:01.708455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 7501
I20260621 02:39:01.732241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 7650
I20260621 02:39:01.746847 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:01.733053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.0.97.3:43879)
0621 02:39:01.733235 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:01.733247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:01.733278 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:39:01.733543 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:01.733549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:01.733629 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:01.733740 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:01.735194 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.736095 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.737551 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:01.738061 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:01.738201 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:01.740854 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:01.740868 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:01.740883 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:01.740917 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:01.744052 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:01.744664 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:01.744678 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:01.744681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:01.744833 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:01.745200 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:01.745207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:01.745210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:01.745600 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:01.745950 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:01.746220 (+ 270us) server_negotiation.cc:299] Negotiation successful
0621 02:39:01.746425 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:39:01.747521 7773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:01.875964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 7650
I20260621 02:39:01.898602 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 6983
I20260621 02:39:01.899788 7155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:02.095913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 6983
I20260621 02:39:02.116065 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:02.115656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:54682)
0621 02:39:02.115827 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:02.115905 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:39:02.117028 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:02.116701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46061 (local address 127.0.0.1:54690)
0621 02:39:02.116834 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:02.116904 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 02:39:02.128242 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 7081
I20260621 02:39:02.129338 7340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:02.246269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 7081
2026-06-21T02:39:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9704 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:39:02 dist-test-slave-jkn7 krb5kdc[7907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:39:02 dist-test-slave-jkn7 krb5kdc[7907](info): set up 2 sockets
Jun 21 02:39:02 dist-test-slave-jkn7 krb5kdc[7907](info): commencing operation
krb5kdc: starting...
W20260621 02:39:03.334221 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:39:03 dist-test-slave-jkn7 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:39:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:39:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260621 02:39:03.548727 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42841
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:03.861660 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:03.861986 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:03.862131 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:03.869215 7920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:39:03.869374 7920 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 02:39:03.869460 7920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:03.869580 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:03.869661 7920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:39:03.869753 7920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:39:03.879268 7920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42841
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:03.880498 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:03.881798 7920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:03.893047 7925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:03.893003 7928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:03.892961 7926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:03.893814 7920 server_base.cc:1034] running on GCE node
I20260621 02:39:03.894507 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:03.895795 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:03.897014 7920 hybrid_clock.cc:648] HybridClock initialized: now 1782009543896958 us; error 28 us; skew 500 ppm
I20260621 02:39:03.897338 7920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:03 dist-test-slave-jkn7 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009543, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:03.900170 7920 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260621 02:39:03.901613 7920 webserver.cc:460] Webserver started at http://127.0.97.62:37891/ using document root <none> and password file <none>
I20260621 02:39:03.902221 7920 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:03.902302 7920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:03.902560 7920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:03.905459 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "c15e384d28374935972cb547ccd3eb9d"
format_stamp: "Formatted at 2026-06-21 02:39:03 on dist-test-slave-jkn7"
server_key: "441f1e54b26582fdc80ed7ac07c1e8b6"
server_key_iv: "c8c1889ba8a7736d176a337a9b6aebe4"
server_key_version: "encryptionkey@0"
I20260621 02:39:03.906173 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "c15e384d28374935972cb547ccd3eb9d"
format_stamp: "Formatted at 2026-06-21 02:39:03 on dist-test-slave-jkn7"
server_key: "441f1e54b26582fdc80ed7ac07c1e8b6"
server_key_iv: "c8c1889ba8a7736d176a337a9b6aebe4"
server_key_version: "encryptionkey@0"
I20260621 02:39:03.910904 7920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:39:03.914050 7935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:03.915313 7920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:39:03.915443 7920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "c15e384d28374935972cb547ccd3eb9d"
format_stamp: "Formatted at 2026-06-21 02:39:03 on dist-test-slave-jkn7"
server_key: "441f1e54b26582fdc80ed7ac07c1e8b6"
server_key_iv: "c8c1889ba8a7736d176a337a9b6aebe4"
server_key_version: "encryptionkey@0"
I20260621 02:39:03.915621 7920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:03.946877 7920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:03.950816 7920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:03.951030 7920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:03.972460 7920 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42841
I20260621 02:39:03.972501 7996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42841 every 8 connection(s)
I20260621 02:39:03.973862 7920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:39:03.976708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 7920
I20260621 02:39:03.976913 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:39:03.977761 388 external_mini_cluster.cc:1428] Setting key 6e35347e984fa8d7e224fd862debc29c
I20260621 02:39:03.978168 7997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:03.992389 7997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d: Bootstrap starting.
Jun 21 02:39:03 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:39:03.996321 7997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:03.997701 7997 log.cc:824] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:04.001051 7997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d: No bootstrap required, opened a new log
I20260621 02:39:04.002198 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:03.979747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.0.97.62:42841)
0621 02:39:03.980436 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:03.980448 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:03.980485 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:39:03.982679 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:03.982702 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:03.983087 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:03.983414 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:03.985124 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:03.986439 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:03.988243 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:03.988958 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:03.989108 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:03.993656 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:03.993667 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:03.993688 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:03.993736 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:03.997680 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:03.998530 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:03.998536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:03.998543 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:03.998706 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:03.999138 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:03.999145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:03.999151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:03.999794 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:04.000071 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:04.000372 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 02:39:04.000798 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":331,"threads_started":1}
I20260621 02:39:04.009903 7997 raft_consensus.cc:348] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15e384d28374935972cb547ccd3eb9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } }
I20260621 02:39:04.010255 7997 raft_consensus.cc:374] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:04.010329 7997 raft_consensus.cc:729] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15e384d28374935972cb547ccd3eb9d, State: Initialized, Role: FOLLOWER
I20260621 02:39:04.010834 7997 consensus_queue.cc:260] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15e384d28374935972cb547ccd3eb9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } }
I20260621 02:39:04.011025 7997 raft_consensus.cc:388] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:39:04.011117 7997 raft_consensus.cc:482] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:39:04.011243 7997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:04.013371 7997 raft_consensus.cc:504] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15e384d28374935972cb547ccd3eb9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } }
I20260621 02:39:04.013892 7997 leader_election.cc:302] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c15e384d28374935972cb547ccd3eb9d; no voters:
I20260621 02:39:04.014472 7997 leader_election.cc:288] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:39:04.014642 8002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:04.015348 8002 raft_consensus.cc:686] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [term 1 LEADER]: Becoming Leader. State: Replica: c15e384d28374935972cb547ccd3eb9d, State: Running, Role: LEADER
I20260621 02:39:04.015872 8002 consensus_queue.cc:237] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15e384d28374935972cb547ccd3eb9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } }
I20260621 02:39:04.016633 7997 sys_catalog.cc:564] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:39:04.018610 8004 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c15e384d28374935972cb547ccd3eb9d. Latest consensus state: current_term: 1 leader_uuid: "c15e384d28374935972cb547ccd3eb9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15e384d28374935972cb547ccd3eb9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } } }
I20260621 02:39:04.019094 8004 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:04.019678 8003 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c15e384d28374935972cb547ccd3eb9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15e384d28374935972cb547ccd3eb9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } } }
I20260621 02:39:04.019997 8003 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:04.021523 8010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:39:04.026561 8010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:39:04.035328 8010 catalog_manager.cc:1347] Generated new cluster ID: 2cccd171e7be4976852f1150f490de01
I20260621 02:39:04.035445 8010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:39:04.061117 8010 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:39:04.062430 8010 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:39:04.071756 8010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d: Generated new TSK 0
I20260621 02:39:04.072536 8010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260621 02:39:04.128067 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42841
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:04.471272 8023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:04.471560 8023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:04.471658 8023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:04.478761 8023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:04.478868 8023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:04.478950 8023 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:39:04.479027 8023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:39:04.488588 8023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:04.489805 8023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:04.491164 8023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:04.503422 8028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:04.503333 8029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:04.503386 8031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:04.504252 8023 server_base.cc:1034] running on GCE node
I20260621 02:39:04.504884 8023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:04.505975 8023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:04.507244 8023 hybrid_clock.cc:648] HybridClock initialized: now 1782009544507170 us; error 44 us; skew 500 ppm
I20260621 02:39:04.507560 8023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:04 dist-test-slave-jkn7 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:04.510915 8023 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260621 02:39:04.512806 8023 webserver.cc:460] Webserver started at http://127.0.97.1:33057/ using document root <none> and password file <none>
I20260621 02:39:04.513590 8023 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:04.513680 8023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:04.513955 8023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:04.516902 8023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "13c35b1c0efc4ec391940f02110c7d1e"
format_stamp: "Formatted at 2026-06-21 02:39:04 on dist-test-slave-jkn7"
server_key: "17de3ccf44776681e2cccc731004bfdd"
server_key_iv: "ec00094bea592fc2b4c5a5b46f638fb0"
server_key_version: "encryptionkey@0"
I20260621 02:39:04.517747 8023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "13c35b1c0efc4ec391940f02110c7d1e"
format_stamp: "Formatted at 2026-06-21 02:39:04 on dist-test-slave-jkn7"
server_key: "17de3ccf44776681e2cccc731004bfdd"
server_key_iv: "ec00094bea592fc2b4c5a5b46f638fb0"
server_key_version: "encryptionkey@0"
I20260621 02:39:04.523005 8023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:39:04.526363 8038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:04.527693 8023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:39:04.527853 8023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "13c35b1c0efc4ec391940f02110c7d1e"
format_stamp: "Formatted at 2026-06-21 02:39:04 on dist-test-slave-jkn7"
server_key: "17de3ccf44776681e2cccc731004bfdd"
server_key_iv: "ec00094bea592fc2b4c5a5b46f638fb0"
server_key_version: "encryptionkey@0"
I20260621 02:39:04.528051 8023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:04.541975 8023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:04.546301 8023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:04.546566 8023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:04.549186 8023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:04.549306 8023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:04.549466 8023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:04.549540 8023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:39:04 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:39:04.573390 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:04.549680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36131 (local address 127.0.97.62:42841)
0621 02:39:04.550135 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:04.550142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:04.551326 (+ 1184us) server_negotiation.cc:407] Connection header received
0621 02:39:04.554329 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:04.554337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:04.554437 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:04.554564 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:04.557043 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:04.558024 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:04.561225 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:04.561729 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:04.561873 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:04.566421 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:04.566426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:04.566429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:04.566460 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:04.569639 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:04.570577 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:04.570582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:04.570585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:04.570731 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:04.571177 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:04.571185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:04.571191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:04.571659 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:04.571821 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:04.572538 (+ 717us) server_negotiation.cc:299] Negotiation successful
0621 02:39:04.572818 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
I20260621 02:39:04.574318 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:04.550303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.97.1:36131)
0621 02:39:04.551134 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:04.551192 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:04.553974 (+ 2782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:04.554869 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:04.554884 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:04.555820 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:04.556840 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:04.556873 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:04.558200 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:04.558206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:04.561014 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:04.561050 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:04.561988 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:04.562002 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:04.562356 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:04.563145 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:04.563184 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:04.566206 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:04.569778 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:04.569786 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:04.569805 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:04.570431 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:04.570861 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:04.570869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:04.570875 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:04.571062 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:04.571817 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:04.571824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:04.572189 (+ 365us) client_negotiation.cc:769] Sending connection context
0621 02:39:04.572408 (+ 219us) client_negotiation.cc:240] Negotiation successful
0621 02:39:04.572764 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":296,"threads_started":1}
I20260621 02:39:04.594942 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:04.580778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54379 (local address 127.0.97.62:42841)
0621 02:39:04.580974 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:04.580983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:04.581552 (+ 569us) server_negotiation.cc:407] Connection header received
0621 02:39:04.581686 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:04.581694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:04.581790 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:04.581918 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:04.582775 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:04.584170 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:04.585788 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:04.586519 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:04.586665 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:04.588840 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:04.588845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:04.588848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:04.588876 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:04.592069 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:04.592986 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:04.592993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:04.592998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:04.593140 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:04.593549 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:04.593554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:04.593557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:04.593800 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:04.593948 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:04.594384 (+ 436us) server_negotiation.cc:299] Negotiation successful
0621 02:39:04.594601 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:39:04.595074 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:04.580777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.97.1:54379)
0621 02:39:04.581447 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:04.581475 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:04.581583 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:04.581959 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:04.581963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:04.582367 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:04.582641 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:04.582649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:04.584304 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:04.584308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:04.585626 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:04.585648 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:04.586735 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:04.586741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:04.586916 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:04.587544 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:04.587561 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:04.588628 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:04.592239 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:04.592247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:04.592252 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:04.592816 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:04.593257 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:04.593263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:04.593266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:04.593421 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:04.593919 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:04.593923 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:04.594121 (+ 198us) client_negotiation.cc:769] Sending connection context
0621 02:39:04.594283 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 02:39:04.594598 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":365,"threads_started":1}
I20260621 02:39:04.607312 8023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44443
I20260621 02:39:04.607342 8158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44443 every 8 connection(s)
I20260621 02:39:04.608736 8023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:39:04.610494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 8023
I20260621 02:39:04.610695 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:39:04.610868 8159 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42841
I20260621 02:39:04.611109 8159 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:04.611418 388 external_mini_cluster.cc:1428] Setting key 3df416e56e5d4cabc8e6e6593a2e95f7
I20260621 02:39:04.611778 8159 heartbeater.cc:511] Master 127.0.97.62:42841 requested a full tablet report, sending...
I20260621 02:39:04.613788 7951 ts_manager.cc:194] Registered new tserver with Master: 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1:44443)
I20260621 02:39:04.615163 7951 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36131
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260621 02:39:04.653918 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42841
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:04.967813 8163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:04.968139 8163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:04.968308 8163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:04.976617 8163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:04.976773 8163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:04.976928 8163 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:39:04.977051 8163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:39:04.986635 8163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:04.987747 8163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:04.988958 8163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:05.001215 8168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:05.001247 8169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:05.001186 8171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:05.001989 8163 server_base.cc:1034] running on GCE node
I20260621 02:39:05.002517 8163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:05.003295 8163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:05.004522 8163 hybrid_clock.cc:648] HybridClock initialized: now 1782009545004453 us; error 47 us; skew 500 ppm
I20260621 02:39:05.004796 8163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:05 dist-test-slave-jkn7 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:05.007808 8163 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260621 02:39:05.009092 8163 webserver.cc:460] Webserver started at http://127.0.97.2:37677/ using document root <none> and password file <none>
I20260621 02:39:05.009742 8163 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:05.009824 8163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:05.010066 8163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:05.012845 8163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "6b1e4a6f17f94606a214371f619cdf14"
format_stamp: "Formatted at 2026-06-21 02:39:05 on dist-test-slave-jkn7"
server_key: "d0f277220b6dc5c9f82ba2c87534acf2"
server_key_iv: "96caa8843017bba4b89cb40e25933ff8"
server_key_version: "encryptionkey@0"
I20260621 02:39:05.013602 8163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6b1e4a6f17f94606a214371f619cdf14"
format_stamp: "Formatted at 2026-06-21 02:39:05 on dist-test-slave-jkn7"
server_key: "d0f277220b6dc5c9f82ba2c87534acf2"
server_key_iv: "96caa8843017bba4b89cb40e25933ff8"
server_key_version: "encryptionkey@0"
I20260621 02:39:05.018479 8163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:39:05.021651 8178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:05.022848 8163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:39:05.022969 8163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "6b1e4a6f17f94606a214371f619cdf14"
format_stamp: "Formatted at 2026-06-21 02:39:05 on dist-test-slave-jkn7"
server_key: "d0f277220b6dc5c9f82ba2c87534acf2"
server_key_iv: "96caa8843017bba4b89cb40e25933ff8"
server_key_version: "encryptionkey@0"
I20260621 02:39:05.023130 8163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:05.045271 8163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:05.049260 8163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:05.049489 8163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:05.052006 8163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:05.052086 8163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:05.052228 8163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:05.052322 8163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:39:05 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:39:05.084540 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.052475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36823 (local address 127.0.97.62:42841)
0621 02:39:05.052668 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.052675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.054091 (+ 1416us) server_negotiation.cc:407] Connection header received
0621 02:39:05.056317 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.056322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.056420 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.056527 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:05.059375 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.060457 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.065095 (+ 4638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.065650 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.065809 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.077153 (+ 11344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:05.077160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:05.077165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:05.077208 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:05.081032 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.082184 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.082190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.082194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.082319 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.082745 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.082750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.082754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.083039 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:05.083187 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.083979 (+ 792us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.084195 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:39:05.085719 8187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.053009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.97.2:36823)
0621 02:39:05.053887 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:05.053944 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:05.055995 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:05.056804 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:05.056813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:05.058224 (+ 1411us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:05.059106 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.059138 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.060751 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.060763 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.064768 (+ 4005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.064800 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.066052 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.066065 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.066367 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.073458 (+ 7091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:05.073515 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:05.076841 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:05.081267 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:05.081319 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:05.081344 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:05.082030 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:05.082428 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:05.082435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:05.082439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:05.082625 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:05.083165 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:05.083181 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:05.083596 (+ 415us) client_negotiation.cc:769] Sending connection context
0621 02:39:05.083860 (+ 264us) client_negotiation.cc:240] Negotiation successful
0621 02:39:05.084413 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":376,"threads_started":1}
I20260621 02:39:05.108103 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.094877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.97.2:39103)
0621 02:39:05.095429 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:05.095444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:05.095599 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:05.095923 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:05.095927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:05.096567 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:05.097067 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.097082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.098441 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.098446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.099797 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.099808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.100503 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.100510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.100683 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.101403 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:05.101418 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:05.102089 (+ 671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:05.104912 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:05.104917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:05.104921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:05.105362 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:05.105709 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:05.105717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:05.105721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:05.105882 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:05.106697 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:05.106704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:05.106949 (+ 245us) client_negotiation.cc:769] Sending connection context
0621 02:39:05.107147 (+ 198us) client_negotiation.cc:240] Negotiation successful
0621 02:39:05.107597 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":299,"threads_started":1}
I20260621 02:39:05.108160 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.094877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39103 (local address 127.0.97.62:42841)
0621 02:39:05.095026 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.095031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.095527 (+ 496us) server_negotiation.cc:407] Connection header received
0621 02:39:05.095735 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.095740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.095806 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.095898 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:05.097216 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.098316 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.099934 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.100331 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.100464 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.102266 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:05.102275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:05.102278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:05.102306 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:05.104782 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.105474 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.105481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.105485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.105595 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.106023 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.106029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.106033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.106552 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:05.106736 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.107633 (+ 897us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.107854 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:39:05.121111 8163 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35885
I20260621 02:39:05.121133 8297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35885 every 8 connection(s)
I20260621 02:39:05.122346 8163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:39:05.124297 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 8163
I20260621 02:39:05.124475 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:39:05.124473 8298 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42841
I20260621 02:39:05.124703 8298 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:05.125046 388 external_mini_cluster.cc:1428] Setting key fad85d082147efe3d20188e25f1e86d8
I20260621 02:39:05.125500 8298 heartbeater.cc:511] Master 127.0.97.62:42841 requested a full tablet report, sending...
I20260621 02:39:05.126941 7952 ts_manager.cc:194] Registered new tserver with Master: 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885)
I20260621 02:39:05.127779 7952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36823
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260621 02:39:05.168632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42841
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:05.488269 8302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:05.488529 8302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:05.488695 8302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:05.496002 8302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:05.496098 8302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:05.496232 8302 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:39:05.496335 8302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:39:05.505846 8302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:05.507022 8302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:05.508332 8302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:05.520990 8307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:05.521028 8310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:05.521057 8308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:05.522083 8302 server_base.cc:1034] running on GCE node
I20260621 02:39:05.522655 8302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:05.523563 8302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:05.524812 8302 hybrid_clock.cc:648] HybridClock initialized: now 1782009545524734 us; error 55 us; skew 500 ppm
I20260621 02:39:05.525104 8302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:05 dist-test-slave-jkn7 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:05.528250 8302 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260621 02:39:05.529772 8302 webserver.cc:460] Webserver started at http://127.0.97.3:34103/ using document root <none> and password file <none>
I20260621 02:39:05.530438 8302 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:05.530521 8302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:05.530814 8302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:05.533762 8302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "ed27d65c42df4b0184372b623fe0f63b"
format_stamp: "Formatted at 2026-06-21 02:39:05 on dist-test-slave-jkn7"
server_key: "808186f80905570c75177ecce2837b5d"
server_key_iv: "6eee5d5d2e98052fdf9554144cd95310"
server_key_version: "encryptionkey@0"
I20260621 02:39:05.534475 8302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed27d65c42df4b0184372b623fe0f63b"
format_stamp: "Formatted at 2026-06-21 02:39:05 on dist-test-slave-jkn7"
server_key: "808186f80905570c75177ecce2837b5d"
server_key_iv: "6eee5d5d2e98052fdf9554144cd95310"
server_key_version: "encryptionkey@0"
I20260621 02:39:05.539354 8302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:39:05.542675 8317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:05.544037 8302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:39:05.544211 8302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "ed27d65c42df4b0184372b623fe0f63b"
format_stamp: "Formatted at 2026-06-21 02:39:05 on dist-test-slave-jkn7"
server_key: "808186f80905570c75177ecce2837b5d"
server_key_iv: "6eee5d5d2e98052fdf9554144cd95310"
server_key_version: "encryptionkey@0"
I20260621 02:39:05.544427 8302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:05.560113 8302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:05.563911 8302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:05.564170 8302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:05.566776 8302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:05.566851 8302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:05.567013 8302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:05.567086 8302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:39:05 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:39:05.590703 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.567211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52579 (local address 127.0.97.62:42841)
0621 02:39:05.567440 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.567449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.569210 (+ 1761us) server_negotiation.cc:407] Connection header received
0621 02:39:05.571200 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.571209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.571305 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.571416 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:05.573500 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.574777 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.577392 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.578138 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.578285 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.583352 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:05.583359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:05.583364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:05.583403 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:05.587174 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.587975 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.587983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.587989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.588190 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.588648 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.588656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.588662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.589100 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:05.589311 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.590125 (+ 814us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.590341 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:39:05.591725 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.567930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.97.3:52579)
0621 02:39:05.569005 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:05.569064 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:05.570917 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:05.571654 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:05.571664 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:05.572410 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:05.573261 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.573287 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.574988 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.574996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.577191 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.577209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.578436 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.578442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.578623 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.579408 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:05.579431 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:05.583049 (+ 3618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:05.587335 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:05.587345 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:05.587360 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:05.587832 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:05.588301 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:05.588305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:05.588307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:05.588518 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:05.589237 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:05.589242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:05.589734 (+ 492us) client_negotiation.cc:769] Sending connection context
0621 02:39:05.590011 (+ 277us) client_negotiation.cc:240] Negotiation successful
0621 02:39:05.590527 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":609,"threads_started":1}
I20260621 02:39:05.613545 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.597035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56367 (local address 127.0.97.62:42841)
0621 02:39:05.597224 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.597231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.597985 (+ 754us) server_negotiation.cc:407] Connection header received
0621 02:39:05.598180 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.598186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.598263 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.598356 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:05.599617 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.600837 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.602625 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.603522 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.603680 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.605447 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:05.605455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:05.605463 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:05.605504 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:05.609950 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.611135 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.611144 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.611150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.611336 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.611837 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.611844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.611850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.612295 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:05.612475 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.612896 (+ 421us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.613154 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:39:05.613847 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.597123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.97.3:56367)
0621 02:39:05.597885 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:05.597900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:05.598063 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:05.598380 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:05.598385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:05.598945 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:05.599385 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.599398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.600972 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.600977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.602446 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.602464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.603738 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.603743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.603855 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.604485 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:05.604502 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:05.605215 (+ 713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:05.610180 (+ 4965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:05.610190 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:05.610197 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:05.610915 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:05.611485 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:05.611493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:05.611499 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:05.611673 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:05.612433 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:05.612437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:05.612616 (+ 179us) client_negotiation.cc:769] Sending connection context
0621 02:39:05.612761 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 02:39:05.613128 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":436,"threads_started":1}
I20260621 02:39:05.617961 8159 heartbeater.cc:503] Master 127.0.97.62:42841 was elected leader, sending a full tablet report...
I20260621 02:39:05.626119 8302 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33635
I20260621 02:39:05.626140 8436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33635 every 8 connection(s)
I20260621 02:39:05.627270 8302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:39:05.627681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 8302
I20260621 02:39:05.627843 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:39:05.628413 388 external_mini_cluster.cc:1428] Setting key aaabacd2232f7d265f3d54e6c8a95177
I20260621 02:39:05.629364 8437 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42841
I20260621 02:39:05.629595 8437 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:05.630278 8437 heartbeater.cc:511] Master 127.0.97.62:42841 requested a full tablet report, sending...
I20260621 02:39:05.631453 7952 ts_manager.cc:194] Registered new tserver with Master: ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635)
I20260621 02:39:05.632315 7952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52579
I20260621 02:39:05.643355 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:39:05 dist-test-slave-jkn7 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009545, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:39:05 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:05 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:39:05.691171 8047 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:05.680972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54576 (local address 127.0.97.62:42841)
0621 02:39:05.681187 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.681196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.681443 (+ 247us) server_negotiation.cc:407] Connection header received
0621 02:39:05.681659 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.681665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.681735 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.681835 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:05.683122 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.683851 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.685080 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.685546 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.685676 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.690897 (+ 5221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54576: BlockingRecv error: recv got EOF from 127.0.0.1:54576 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 21 02:39:05 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009545, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:39:05.710016 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.696923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.0.97.62:42841)
0621 02:39:05.697125 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.697134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.697439 (+ 305us) server_negotiation.cc:407] Connection header received
0621 02:39:05.697624 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.697629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.697712 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.697811 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:05.698878 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.699588 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.700964 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.701425 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.701546 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.705012 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:05.705018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:05.705023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:05.705064 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:05.707676 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.708337 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.708345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.708348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.708483 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:05.708810 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:05.708813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:05.708815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:05.709058 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:05.709258 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.709614 (+ 356us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.709800 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:39:05.713850 7952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:39:05.720724 7952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:39:05.753924 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.732671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33635 (local address 127.0.0.1:48648)
0621 02:39:05.745419 (+ 12748us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:05.745438 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:05.745633 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:05.746520 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:05.746524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:05.746546 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:05.746946 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.746963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.749048 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.749054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.751231 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.751248 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.752885 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.752891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.753076 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.753142 (+ 66us) client_negotiation.cc:769] Sending connection context
0621 02:39:05.753311 (+ 169us) client_negotiation.cc:240] Negotiation successful
0621 02:39:05.753486 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12644,"mutex_wait_us":96,"thread_start_us":10864,"threads_started":1}
I20260621 02:39:05.754256 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.732559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44443 (local address 127.0.0.1:42384)
0621 02:39:05.743932 (+ 11373us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:05.743971 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:05.744166 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:05.746122 (+ 1956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:05.746129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:05.746157 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:05.746656 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.746668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.748873 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.748879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.751883 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.751896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.753323 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.753330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.753493 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.753526 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:39:05.753612 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:39:05.753739 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11100,"mutex_wait_us":265,"spinlock_wait_cycles":22144,"thread_start_us":12592,"threads_started":1}
I20260621 02:39:05.754546 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.732753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.0.97.1:44443)
0621 02:39:05.745441 (+ 12688us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.745451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.745482 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:39:05.745586 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.745594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.745856 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.746865 (+ 1009us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:05.747073 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.748717 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.752060 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.753095 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.753223 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.753447 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.753675 (+ 228us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.753829 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12534,"thread_start_us":12395,"threads_started":1}
I20260621 02:39:05.754685 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.733008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48648 (local address 127.0.97.3:33635)
0621 02:39:05.744821 (+ 11813us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.744830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.745540 (+ 710us) server_negotiation.cc:407] Connection header received
0621 02:39:05.745790 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.745796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.746013 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.746166 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:05.747414 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.748937 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.751418 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.752698 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.752827 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.753062 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.754213 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.754376 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11614,"thread_start_us":11811,"threads_started":1}
I20260621 02:39:05.755298 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.735413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.97.2:35885)
0621 02:39:05.745744 (+ 10331us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.745756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.746310 (+ 554us) server_negotiation.cc:407] Connection header received
0621 02:39:05.746525 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.746531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.746702 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.746877 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:05.747485 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.748697 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.752035 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.753384 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.753505 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.753691 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.754959 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.755073 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10161,"thread_start_us":10055,"threads_started":1}
I20260621 02:39:05.755767 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.734805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35885 (local address 127.0.0.1:43674)
0621 02:39:05.746196 (+ 11391us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:05.746211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:05.746369 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:05.746868 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:05.746872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:05.746901 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:05.747309 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.747322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.748881 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.748885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.751882 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.751896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.753825 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.753832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.754028 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.754062 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:39:05.754395 (+ 333us) client_negotiation.cc:240] Negotiation successful
0621 02:39:05.754529 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11283,"mutex_wait_us":225,"spinlock_wait_cycles":768,"thread_start_us":11122,"threads_started":1}
I20260621 02:39:05.758150 8367 tablet_service.cc:1467] Processing CreateTablet for tablet 04df37eb8bcc4543b906055e7ae5deb7 (DEFAULT_TABLE table=test-table [id=0d157747034741e5bcab468b10037c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:05.758503 8232 tablet_service.cc:1467] Processing CreateTablet for tablet 04df37eb8bcc4543b906055e7ae5deb7 (DEFAULT_TABLE table=test-table [id=0d157747034741e5bcab468b10037c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:05.759577 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04df37eb8bcc4543b906055e7ae5deb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:05.759630 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04df37eb8bcc4543b906055e7ae5deb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:05.759516 8089 tablet_service.cc:1467] Processing CreateTablet for tablet 04df37eb8bcc4543b906055e7ae5deb7 (DEFAULT_TABLE table=test-table [id=0d157747034741e5bcab468b10037c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:05.760807 8089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04df37eb8bcc4543b906055e7ae5deb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:05.770072 8461 tablet_bootstrap.cc:492] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: Bootstrap starting.
I20260621 02:39:05.773313 8461 tablet_bootstrap.cc:654] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:05.774435 8461 log.cc:824] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:05.777251 8461 tablet_bootstrap.cc:492] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: No bootstrap required, opened a new log
I20260621 02:39:05.777558 8461 ts_tablet_manager.cc:1397] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 02:39:05.777946 8463 tablet_bootstrap.cc:492] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: Bootstrap starting.
I20260621 02:39:05.780579 8463 tablet_bootstrap.cc:654] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:05.780885 8462 tablet_bootstrap.cc:492] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: Bootstrap starting.
I20260621 02:39:05.781724 8463 log.cc:824] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:05.783975 8463 tablet_bootstrap.cc:492] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: No bootstrap required, opened a new log
I20260621 02:39:05.784217 8463 ts_tablet_manager.cc:1397] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260621 02:39:05.784677 8462 tablet_bootstrap.cc:654] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:05.785889 8462 log.cc:824] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:05.786614 8461 raft_consensus.cc:348] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } }
I20260621 02:39:05.786949 8461 raft_consensus.cc:374] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:05.787065 8461 raft_consensus.cc:729] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1e4a6f17f94606a214371f619cdf14, State: Initialized, Role: FOLLOWER
I20260621 02:39:05.787701 8461 consensus_queue.cc:260] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } }
I20260621 02:39:05.788771 8298 heartbeater.cc:503] Master 127.0.97.62:42841 was elected leader, sending a full tablet report...
I20260621 02:39:05.788852 8462 tablet_bootstrap.cc:492] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: No bootstrap required, opened a new log
I20260621 02:39:05.789126 8462 ts_tablet_manager.cc:1397] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260621 02:39:05.789425 8461 ts_tablet_manager.cc:1428] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260621 02:39:05.795686 8463 raft_consensus.cc:348] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } }
I20260621 02:39:05.796103 8463 raft_consensus.cc:374] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:05.796200 8463 raft_consensus.cc:729] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c35b1c0efc4ec391940f02110c7d1e, State: Initialized, Role: FOLLOWER
I20260621 02:39:05.796800 8463 consensus_queue.cc:260] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } }
I20260621 02:39:05.798565 8463 ts_tablet_manager.cc:1428] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260621 02:39:05.798431 8462 raft_consensus.cc:348] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } }
I20260621 02:39:05.798830 8462 raft_consensus.cc:374] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:05.798935 8462 raft_consensus.cc:729] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed27d65c42df4b0184372b623fe0f63b, State: Initialized, Role: FOLLOWER
I20260621 02:39:05.799448 8462 consensus_queue.cc:260] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } }
I20260621 02:39:05.800395 8437 heartbeater.cc:503] Master 127.0.97.62:42841 was elected leader, sending a full tablet report...
I20260621 02:39:05.800958 8462 ts_tablet_manager.cc:1428] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260621 02:39:05.823154 8468 raft_consensus.cc:482] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:39:05.823450 8468 raft_consensus.cc:504] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } }
I20260621 02:39:05.825451 8468 leader_election.cc:288] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635), 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885)
I20260621 02:39:05.834637 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.825848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44087 (local address 127.0.97.3:33635)
0621 02:39:05.826160 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.826168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.826836 (+ 668us) server_negotiation.cc:407] Connection header received
0621 02:39:05.827040 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.827046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.827122 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.827216 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:05.828059 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.829182 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.832229 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.833511 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.833653 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.833758 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.834250 (+ 492us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.834383 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260621 02:39:05.834654 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.825954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51267 (local address 127.0.97.2:35885)
0621 02:39:05.826125 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.826131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.826881 (+ 750us) server_negotiation.cc:407] Connection header received
0621 02:39:05.827040 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.827046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.827126 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.827216 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:05.828059 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.829183 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.832229 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.833538 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.833663 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.833758 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.834250 (+ 492us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.834383 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 02:39:05.834854 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.825658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33635 (local address 127.0.97.1:44087)
0621 02:39:05.826703 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:05.826727 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:05.826877 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:05.827263 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:05.827269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:05.827333 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:05.827897 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.827912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.829341 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.829347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.832064 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.832072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.833826 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.833833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.834061 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.834110 (+ 49us) client_negotiation.cc:769] Sending connection context
0621 02:39:05.834200 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 02:39:05.834399 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":218,"thread_start_us":631,"threads_started":1}
I20260621 02:39:05.835171 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.825789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35885 (local address 127.0.97.1:51267)
0621 02:39:05.826725 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:05.826739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:05.826910 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:05.827329 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:05.827334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:05.827352 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:05.827865 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.827882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.829342 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.829347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.832056 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:05.832072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.833813 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.833820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:05.834052 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.834102 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 02:39:05.834200 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:39:05.834399 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":315,"spinlock_wait_cycles":34176,"thread_start_us":571,"threads_started":1}
I20260621 02:39:05.835830 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04df37eb8bcc4543b906055e7ae5deb7" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed27d65c42df4b0184372b623fe0f63b" is_pre_election: true
I20260621 02:39:05.835984 8253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04df37eb8bcc4543b906055e7ae5deb7" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e4a6f17f94606a214371f619cdf14" is_pre_election: true
I20260621 02:39:05.836326 8392 raft_consensus.cc:2454] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 0.
I20260621 02:39:05.836448 8253 raft_consensus.cc:2454] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 0.
I20260621 02:39:05.836990 8042 leader_election.cc:302] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13c35b1c0efc4ec391940f02110c7d1e, 6b1e4a6f17f94606a214371f619cdf14, ed27d65c42df4b0184372b623fe0f63b; no voters:
I20260621 02:39:05.837404 8468 raft_consensus.cc:2781] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:39:05.837541 8468 raft_consensus.cc:482] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:39:05.837656 8468 raft_consensus.cc:3037] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:05.839753 8468 raft_consensus.cc:504] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } }
I20260621 02:39:05.840384 8468 leader_election.cc:288] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 election: Requested vote from peers ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635), 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885)
I20260621 02:39:05.840785 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04df37eb8bcc4543b906055e7ae5deb7" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed27d65c42df4b0184372b623fe0f63b"
I20260621 02:39:05.840847 8253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04df37eb8bcc4543b906055e7ae5deb7" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e4a6f17f94606a214371f619cdf14"
I20260621 02:39:05.840974 8392 raft_consensus.cc:3037] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:05.841046 8253 raft_consensus.cc:3037] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:05.843178 8392 raft_consensus.cc:2454] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 1.
I20260621 02:39:05.843225 8253 raft_consensus.cc:2454] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 1.
I20260621 02:39:05.843564 8042 leader_election.cc:302] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13c35b1c0efc4ec391940f02110c7d1e, 6b1e4a6f17f94606a214371f619cdf14, ed27d65c42df4b0184372b623fe0f63b; no voters:
I20260621 02:39:05.843837 8468 raft_consensus.cc:2781] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:05.844239 8468 raft_consensus.cc:686] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 LEADER]: Becoming Leader. State: Replica: 13c35b1c0efc4ec391940f02110c7d1e, State: Running, Role: LEADER
I20260621 02:39:05.844666 8468 consensus_queue.cc:237] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } }
I20260621 02:39:05.849035 7951 catalog_manager.cc:5495] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "13c35b1c0efc4ec391940f02110c7d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } health_report { overall_health: UNKNOWN } } }
W20260621 02:39:05.861013 8160 tablet.cc:2369] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:39:05.874291 8299 tablet.cc:2369] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:39:05.876091 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:05.870912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.0.97.1:44443)
0621 02:39:05.871119 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:05.871126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:05.871260 (+ 134us) server_negotiation.cc:407] Connection header received
0621 02:39:05.871491 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:05.871495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:05.871563 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:05.871710 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:05.872061 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.872737 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.874080 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:05.874516 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:05.874647 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:05.875022 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:05.875494 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:05.875657 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:05.875708 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 02:39:05.875779 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260621 02:39:05.878942 8438 tablet.cc:2369] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:39:05.883622 8392 raft_consensus.cc:1261] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Refusing update from remote peer 13c35b1c0efc4ec391940f02110c7d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:05.883638 8253 raft_consensus.cc:1261] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Refusing update from remote peer 13c35b1c0efc4ec391940f02110c7d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:05.884577 8472 consensus_queue.cc:1035] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:05.884902 8468 consensus_queue.cc:1035] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:05.912351 7951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:05.912684 7951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:05.916267 7951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d: Sending DeleteTablet for 3 replicas of tablet 04df37eb8bcc4543b906055e7ae5deb7
I20260621 02:39:05.917299 8089 tablet_service.cc:1514] Processing DeleteTablet for tablet 04df37eb8bcc4543b906055e7ae5deb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42384
I20260621 02:39:05.917454 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet 04df37eb8bcc4543b906055e7ae5deb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48648
I20260621 02:39:05.917677 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet 04df37eb8bcc4543b906055e7ae5deb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43674
I20260621 02:39:05.919734 7951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:39:05.921111 7951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:39:05.931231 8366 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc63bd5e67446e58825bf0af835a442 (DEFAULT_TABLE table=test-table [id=7737ec3cafb14847bf1a36a9f2b82140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:05.931456 8088 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc63bd5e67446e58825bf0af835a442 (DEFAULT_TABLE table=test-table [id=7737ec3cafb14847bf1a36a9f2b82140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:05.931792 8366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc63bd5e67446e58825bf0af835a442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:05.931826 8231 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc63bd5e67446e58825bf0af835a442 (DEFAULT_TABLE table=test-table [id=7737ec3cafb14847bf1a36a9f2b82140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:05.932026 8088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc63bd5e67446e58825bf0af835a442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:05.932508 8231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc63bd5e67446e58825bf0af835a442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:05.938359 8462 tablet_bootstrap.cc:492] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: Bootstrap starting.
I20260621 02:39:05.938575 8463 tablet_bootstrap.cc:492] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: Bootstrap starting.
I20260621 02:39:05.939939 8462 tablet_bootstrap.cc:654] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:05.940212 8463 tablet_bootstrap.cc:654] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:05.941668 8461 tablet_bootstrap.cc:492] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: Bootstrap starting.
I20260621 02:39:05.943744 8461 tablet_bootstrap.cc:654] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:05.957100 8462 tablet_bootstrap.cc:492] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: No bootstrap required, opened a new log
I20260621 02:39:05.957265 8462 ts_tablet_manager.cc:1397] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.006s
I20260621 02:39:05.957278 8461 tablet_bootstrap.cc:492] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: No bootstrap required, opened a new log
I20260621 02:39:05.957559 8461 ts_tablet_manager.cc:1397] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.009s
I20260621 02:39:05.958585 8461 raft_consensus.cc:348] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:05.958943 8461 raft_consensus.cc:374] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:05.959110 8461 raft_consensus.cc:729] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1e4a6f17f94606a214371f619cdf14, State: Initialized, Role: FOLLOWER
I20260621 02:39:05.958981 8462 raft_consensus.cc:348] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:05.959215 8462 raft_consensus.cc:374] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:05.959314 8462 raft_consensus.cc:729] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed27d65c42df4b0184372b623fe0f63b, State: Initialized, Role: FOLLOWER
I20260621 02:39:05.959614 8462 consensus_queue.cc:260] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:05.959617 8461 consensus_queue.cc:260] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:05.960295 8462 ts_tablet_manager.cc:1428] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 02:39:05.960937 8486 tablet_replica.cc:330] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: stopping tablet replica
I20260621 02:39:05.961402 8486 raft_consensus.cc:2229] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:05.961903 8486 raft_consensus.cc:2258] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:05.965032 8461 ts_tablet_manager.cc:1428] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260621 02:39:05.965785 8487 tablet_replica.cc:330] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: stopping tablet replica
I20260621 02:39:05.966526 8485 tablet_replica.cc:330] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: stopping tablet replica
I20260621 02:39:05.966958 8485 raft_consensus.cc:2229] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:39:05.967628 8485 raft_consensus.cc:2258] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:05.968983 8487 raft_consensus.cc:2229] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:05.969641 8487 raft_consensus.cc:2258] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:05.970290 8486 ts_tablet_manager.cc:1905] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:05.971374 8463 tablet_bootstrap.cc:492] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: No bootstrap required, opened a new log
I20260621 02:39:05.971590 8463 ts_tablet_manager.cc:1397] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.008s
I20260621 02:39:05.972560 8487 ts_tablet_manager.cc:1905] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:05.972504 8463 raft_consensus.cc:348] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:05.972867 8463 raft_consensus.cc:374] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:05.973014 8463 raft_consensus.cc:729] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c35b1c0efc4ec391940f02110c7d1e, State: Initialized, Role: FOLLOWER
I20260621 02:39:05.973407 8463 consensus_queue.cc:260] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:05.974063 8463 ts_tablet_manager.cc:1428] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 02:39:05.980518 8487 ts_tablet_manager.cc:1918] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:05.980721 8487 log.cc:1196] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/04df37eb8bcc4543b906055e7ae5deb7
I20260621 02:39:05.981252 8487 ts_tablet_manager.cc:1939] T 04df37eb8bcc4543b906055e7ae5deb7 P 6b1e4a6f17f94606a214371f619cdf14: Deleting consensus metadata
I20260621 02:39:05.983084 7938 catalog_manager.cc:4841] TS 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885): tablet 04df37eb8bcc4543b906055e7ae5deb7 (table test-table [id=0d157747034741e5bcab468b10037c33]) successfully deleted
I20260621 02:39:05.986044 8486 ts_tablet_manager.cc:1918] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:05.986222 8486 log.cc:1196] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/04df37eb8bcc4543b906055e7ae5deb7
I20260621 02:39:05.986663 8486 ts_tablet_manager.cc:1939] T 04df37eb8bcc4543b906055e7ae5deb7 P ed27d65c42df4b0184372b623fe0f63b: Deleting consensus metadata
I20260621 02:39:05.988171 7939 catalog_manager.cc:4841] TS ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635): tablet 04df37eb8bcc4543b906055e7ae5deb7 (table test-table [id=0d157747034741e5bcab468b10037c33]) successfully deleted
I20260621 02:39:05.989562 8485 ts_tablet_manager.cc:1905] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:05.994799 8485 ts_tablet_manager.cc:1918] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:05.994988 8485 log.cc:1196] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/04df37eb8bcc4543b906055e7ae5deb7
I20260621 02:39:05.995545 8485 ts_tablet_manager.cc:1939] T 04df37eb8bcc4543b906055e7ae5deb7 P 13c35b1c0efc4ec391940f02110c7d1e: Deleting consensus metadata
I20260621 02:39:05.997151 7936 catalog_manager.cc:4841] TS 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1:44443): tablet 04df37eb8bcc4543b906055e7ae5deb7 (table test-table [id=0d157747034741e5bcab468b10037c33]) successfully deleted
I20260621 02:39:06.114094 8480 raft_consensus.cc:482] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:39:06.114317 8480 raft_consensus.cc:504] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.115141 8480 leader_election.cc:288] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885), ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635)
I20260621 02:39:06.115643 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc63bd5e67446e58825bf0af835a442" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed27d65c42df4b0184372b623fe0f63b" is_pre_election: true
I20260621 02:39:06.115667 8253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc63bd5e67446e58825bf0af835a442" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e4a6f17f94606a214371f619cdf14" is_pre_election: true
I20260621 02:39:06.115866 8392 raft_consensus.cc:2454] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 0.
I20260621 02:39:06.115936 8253 raft_consensus.cc:2454] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 0.
I20260621 02:39:06.116381 8042 leader_election.cc:302] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c35b1c0efc4ec391940f02110c7d1e, ed27d65c42df4b0184372b623fe0f63b; no voters:
I20260621 02:39:06.116703 8480 raft_consensus.cc:2781] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:39:06.116828 8480 raft_consensus.cc:482] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:39:06.116921 8480 raft_consensus.cc:3037] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:06.120180 8480 raft_consensus.cc:504] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.120792 8480 leader_election.cc:288] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 election: Requested vote from peers 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885), ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635)
I20260621 02:39:06.121367 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc63bd5e67446e58825bf0af835a442" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed27d65c42df4b0184372b623fe0f63b"
I20260621 02:39:06.121592 8392 raft_consensus.cc:3037] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:06.121737 8253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc63bd5e67446e58825bf0af835a442" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e4a6f17f94606a214371f619cdf14"
I20260621 02:39:06.121920 8253 raft_consensus.cc:3037] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:06.124330 8392 raft_consensus.cc:2454] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 1.
I20260621 02:39:06.124805 8042 leader_election.cc:302] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c35b1c0efc4ec391940f02110c7d1e, ed27d65c42df4b0184372b623fe0f63b; no voters:
I20260621 02:39:06.124886 8253 raft_consensus.cc:2454] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 1.
I20260621 02:39:06.125461 8480 raft_consensus.cc:2781] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:06.125645 8480 raft_consensus.cc:686] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 LEADER]: Becoming Leader. State: Replica: 13c35b1c0efc4ec391940f02110c7d1e, State: Running, Role: LEADER
I20260621 02:39:06.125945 8480 consensus_queue.cc:237] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.131291 7952 catalog_manager.cc:5495] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "13c35b1c0efc4ec391940f02110c7d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } health_report { overall_health: UNKNOWN } } }
Jun 21 02:39:06 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009543, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:39:06.182848 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.168188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.0.97.62:42841)
0621 02:39:06.168462 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:06.168471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:06.168865 (+ 394us) server_negotiation.cc:407] Connection header received
0621 02:39:06.169032 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:06.169035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:06.169092 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:06.169170 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:06.170630 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.171375 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.173221 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.173731 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.173828 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.177728 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:06.177732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:06.177735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:06.177769 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:06.180251 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:06.180913 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:06.180921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:06.180924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:06.181039 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:06.181396 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:06.181403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:06.181407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:06.181658 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:06.181804 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:06.182188 (+ 384us) server_negotiation.cc:299] Negotiation successful
0621 02:39:06.182488 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":113}
I20260621 02:39:06.182868 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.168102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.0.1:54580)
0621 02:39:06.168749 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:06.168775 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:06.168918 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:06.169227 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:06.169232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:06.170142 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:06.170477 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.170489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.171499 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.171505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.173060 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.173074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.173904 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.173910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.174079 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.174778 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:06.174800 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:06.177539 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:06.180398 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:06.180411 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:06.180419 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:06.180806 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:06.181144 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:06.181148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:06.181152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:06.181275 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:06.181784 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:06.181792 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:06.181945 (+ 153us) client_negotiation.cc:769] Sending connection context
0621 02:39:06.182096 (+ 151us) client_negotiation.cc:240] Negotiation successful
0621 02:39:06.182373 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":379,"threads_started":1}
I20260621 02:39:06.188303 7952 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54580:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 02:39:06.189327 7952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:39:06.196396 8089 tablet_service.cc:1467] Processing CreateTablet for tablet e1a030fcad03479e80cc9bde12152b29 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bafddcba1a4548d5bcd6d2812109ffc3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:39:06.196632 8232 tablet_service.cc:1467] Processing CreateTablet for tablet e1a030fcad03479e80cc9bde12152b29 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bafddcba1a4548d5bcd6d2812109ffc3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:39:06.197026 8089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1a030fcad03479e80cc9bde12152b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:06.197124 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1a030fcad03479e80cc9bde12152b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:06.204488 8461 tablet_bootstrap.cc:492] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: Bootstrap starting.
I20260621 02:39:06.204495 8463 tablet_bootstrap.cc:492] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: Bootstrap starting.
I20260621 02:39:06.204697 8367 tablet_service.cc:1467] Processing CreateTablet for tablet e1a030fcad03479e80cc9bde12152b29 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bafddcba1a4548d5bcd6d2812109ffc3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:39:06.205274 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1a030fcad03479e80cc9bde12152b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:06.206233 8463 tablet_bootstrap.cc:654] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:06.206226 8461 tablet_bootstrap.cc:654] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:06.213030 8462 tablet_bootstrap.cc:492] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: Bootstrap starting.
I20260621 02:39:06.215049 8462 tablet_bootstrap.cc:654] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:06.217792 8463 tablet_bootstrap.cc:492] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: No bootstrap required, opened a new log
I20260621 02:39:06.217983 8462 tablet_bootstrap.cc:492] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: No bootstrap required, opened a new log
I20260621 02:39:06.218003 8463 ts_tablet_manager.cc:1397] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260621 02:39:06.218160 8462 ts_tablet_manager.cc:1397] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260621 02:39:06.218845 8463 raft_consensus.cc:348] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.218921 8462 raft_consensus.cc:348] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.219094 8463 raft_consensus.cc:374] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:06.219173 8463 raft_consensus.cc:729] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c35b1c0efc4ec391940f02110c7d1e, State: Initialized, Role: FOLLOWER
I20260621 02:39:06.219177 8462 raft_consensus.cc:374] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:06.219251 8462 raft_consensus.cc:729] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed27d65c42df4b0184372b623fe0f63b, State: Initialized, Role: FOLLOWER
I20260621 02:39:06.219408 8463 consensus_queue.cc:260] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.219617 8462 consensus_queue.cc:260] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.220090 8463 ts_tablet_manager.cc:1428] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 02:39:06.219959 8480 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.220259 8462 ts_tablet_manager.cc:1428] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 02:39:06.220299 8480 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:39:06.220846 8469 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.221145 8469 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:39:06.221508 8461 tablet_bootstrap.cc:492] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: No bootstrap required, opened a new log
I20260621 02:39:06.221735 8461 ts_tablet_manager.cc:1397] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.002s
I20260621 02:39:06.222575 8461 raft_consensus.cc:348] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.222915 8461 raft_consensus.cc:374] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:06.223042 8461 raft_consensus.cc:729] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1e4a6f17f94606a214371f619cdf14, State: Initialized, Role: FOLLOWER
I20260621 02:39:06.223366 8461 consensus_queue.cc:260] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.223909 8467 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.224262 8467 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:39:06.224757 8461 ts_tablet_manager.cc:1428] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260621 02:39:06.430393 8469 raft_consensus.cc:482] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:39:06.430657 8469 raft_consensus.cc:504] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.432222 8469 leader_election.cc:288] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1:44443), 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885)
I20260621 02:39:06.450570 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.432610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40941 (local address 127.0.97.2:35885)
0621 02:39:06.442956 (+ 10346us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:06.442965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:06.443326 (+ 361us) server_negotiation.cc:407] Connection header received
0621 02:39:06.443579 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:06.443587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:06.443679 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:06.443792 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:06.444260 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.445380 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.448169 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.449434 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.449566 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.449664 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:06.450108 (+ 444us) server_negotiation.cc:299] Negotiation successful
0621 02:39:06.450255 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10167,"thread_start_us":10029,"threads_started":1}
I20260621 02:39:06.450572 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.432613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57375 (local address 127.0.97.1:44443)
0621 02:39:06.444026 (+ 11413us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:06.444030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:06.444044 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:39:06.444095 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:06.444098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:06.444148 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:06.444267 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:06.444651 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.445633 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.448169 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.449425 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.449575 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.449671 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:06.450107 (+ 436us) server_negotiation.cc:299] Negotiation successful
0621 02:39:06.450252 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11319,"thread_start_us":10786,"threads_started":1}
I20260621 02:39:06.450743 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.432504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35885 (local address 127.0.97.3:40941)
0621 02:39:06.443185 (+ 10681us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:06.443221 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:06.443424 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:06.443829 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:06.443834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:06.443866 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:06.444116 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.444125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.445569 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.445574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.448006 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.448022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.449634 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.449639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.449885 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.449936 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 02:39:06.450040 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 02:39:06.450242 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10517,"mutex_wait_us":231,"thread_start_us":11097,"threads_started":1}
I20260621 02:39:06.451265 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.432522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44443 (local address 127.0.97.3:57375)
0621 02:39:06.443832 (+ 11310us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:06.443849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:06.443958 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:06.444287 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:06.444290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:06.444310 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:06.444513 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.444520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.445768 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.445774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.448005 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.448022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.449631 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.449637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.449881 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.449941 (+ 60us) client_negotiation.cc:769] Sending connection context
0621 02:39:06.450040 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 02:39:06.450242 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11201,"mutex_wait_us":158,"spinlock_wait_cycles":19712,"thread_start_us":10475,"threads_started":1}
I20260621 02:39:06.451428 8253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a030fcad03479e80cc9bde12152b29" candidate_uuid: "ed27d65c42df4b0184372b623fe0f63b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e4a6f17f94606a214371f619cdf14" is_pre_election: true
I20260621 02:39:06.451740 8253 raft_consensus.cc:2454] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed27d65c42df4b0184372b623fe0f63b in term 0.
I20260621 02:39:06.452167 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a030fcad03479e80cc9bde12152b29" candidate_uuid: "ed27d65c42df4b0184372b623fe0f63b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c35b1c0efc4ec391940f02110c7d1e" is_pre_election: true
I20260621 02:39:06.452255 8320 leader_election.cc:302] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1e4a6f17f94606a214371f619cdf14, ed27d65c42df4b0184372b623fe0f63b; no voters:
I20260621 02:39:06.452540 8114 raft_consensus.cc:2454] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed27d65c42df4b0184372b623fe0f63b in term 0.
I20260621 02:39:06.452632 8469 raft_consensus.cc:2781] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:39:06.452744 8469 raft_consensus.cc:482] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:39:06.452863 8469 raft_consensus.cc:3037] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:06.454968 8469 raft_consensus.cc:504] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.455533 8469 leader_election.cc:288] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [CANDIDATE]: Term 1 election: Requested vote from peers 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1:44443), 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885)
I20260621 02:39:06.455893 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a030fcad03479e80cc9bde12152b29" candidate_uuid: "ed27d65c42df4b0184372b623fe0f63b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c35b1c0efc4ec391940f02110c7d1e"
I20260621 02:39:06.456084 8114 raft_consensus.cc:3037] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:06.456018 8253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a030fcad03479e80cc9bde12152b29" candidate_uuid: "ed27d65c42df4b0184372b623fe0f63b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e4a6f17f94606a214371f619cdf14"
I20260621 02:39:06.456251 8253 raft_consensus.cc:3037] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:06.458145 8253 raft_consensus.cc:2454] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed27d65c42df4b0184372b623fe0f63b in term 1.
I20260621 02:39:06.458295 8114 raft_consensus.cc:2454] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed27d65c42df4b0184372b623fe0f63b in term 1.
I20260621 02:39:06.458504 8320 leader_election.cc:302] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1e4a6f17f94606a214371f619cdf14, ed27d65c42df4b0184372b623fe0f63b; no voters:
I20260621 02:39:06.458817 8469 raft_consensus.cc:2781] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:06.459170 8469 raft_consensus.cc:686] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [term 1 LEADER]: Becoming Leader. State: Replica: ed27d65c42df4b0184372b623fe0f63b, State: Running, Role: LEADER
I20260621 02:39:06.459549 8469 consensus_queue.cc:237] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.461452 8505 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: TxnStatusTablet state changed. Reason: New leader ed27d65c42df4b0184372b623fe0f63b. Latest consensus state: current_term: 1 leader_uuid: "ed27d65c42df4b0184372b623fe0f63b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.461829 8505 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:39:06.462447 8507 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 02:39:06.462675 8507 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 02:39:06.463130 7952 catalog_manager.cc:5495] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b reported cstate change: term changed from 0 to 1, leader changed from <none> to ed27d65c42df4b0184372b623fe0f63b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ed27d65c42df4b0184372b623fe0f63b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } health_report { overall_health: HEALTHY } } }
I20260621 02:39:06.515673 8480 consensus_queue.cc:1035] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:06.516330 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.510101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33635 (local address 127.0.0.1:48654)
0621 02:39:06.510577 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:06.510601 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:06.510741 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:06.511217 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:06.511222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:06.511233 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:39:06.511460 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.511472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.512554 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.512560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.513635 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.513646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.514367 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.514372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.514526 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.514575 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:39:06.515755 (+ 1180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:39:06.515775 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 02:39:06.515863 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 02:39:06.516005 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260621 02:39:06.516783 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.510251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.0.97.3:33635)
0621 02:39:06.510786 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:06.510792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:06.510812 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:39:06.510932 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:06.510937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:06.511041 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:06.511203 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:06.511627 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.512420 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.513774 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.514225 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.514359 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.514846 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:06.515640 (+ 794us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:06.515781 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:06.516040 (+ 259us) server_negotiation.cc:299] Negotiation successful
0621 02:39:06.516119 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":624,"threads_started":1}
I20260621 02:39:06.521530 8253 raft_consensus.cc:1261] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Refusing update from remote peer ed27d65c42df4b0184372b623fe0f63b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:06.522006 8114 raft_consensus.cc:1261] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Refusing update from remote peer ed27d65c42df4b0184372b623fe0f63b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:06.522769 8469 consensus_queue.cc:1035] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:06.523630 8469 consensus_queue.cc:1035] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:06.525800 8472 consensus_queue.cc:1035] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:06.546339 8472 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: TxnStatusTablet state changed. Reason: New leader ed27d65c42df4b0184372b623fe0f63b. Latest consensus state: current_term: 1 leader_uuid: "ed27d65c42df4b0184372b623fe0f63b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.546653 8472 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:39:06.548991 8512 mvcc.cc:205] Tried to move back new op lower bound from 7299111102545670144 to 7299111102301552640. Current Snapshot: MvccSnapshot[applied={T|T < 7299111102545670144}]
I20260621 02:39:06.549811 8513 mvcc.cc:205] Tried to move back new op lower bound from 7299111102545670144 to 7299111102301552640. Current Snapshot: MvccSnapshot[applied={T|T < 7299111102545670144}]
I20260621 02:39:06.558139 8516 mvcc.cc:205] Tried to move back new op lower bound from 7299111102545670144 to 7299111102301552640. Current Snapshot: MvccSnapshot[applied={T|T < 7299111102545670144}]
I20260621 02:39:06.559391 8505 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed27d65c42df4b0184372b623fe0f63b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.559746 8505 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:39:06.560242 8467 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: TxnStatusTablet state changed. Reason: New leader ed27d65c42df4b0184372b623fe0f63b. Latest consensus state: current_term: 1 leader_uuid: "ed27d65c42df4b0184372b623fe0f63b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.560536 8467 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:39:06.560978 8517 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed27d65c42df4b0184372b623fe0f63b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.561323 8517 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P 6b1e4a6f17f94606a214371f619cdf14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:39:06.562404 8472 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed27d65c42df4b0184372b623fe0f63b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.562644 8472 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P 13c35b1c0efc4ec391940f02110c7d1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:39:06.563752 8505 tablet_replica.cc:439] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed27d65c42df4b0184372b623fe0f63b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } } }
I20260621 02:39:06.563966 8505 tablet_replica.cc:442] T e1a030fcad03479e80cc9bde12152b29 P ed27d65c42df4b0184372b623fe0f63b: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:39:06.580721 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.575166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33635 (local address 127.0.97.1:43231)
0621 02:39:06.575703 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:06.575719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:06.575878 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:06.576198 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:06.576201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:06.576210 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:39:06.576420 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.576426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.577860 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.577864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.578864 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.578874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.579773 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.579777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.579891 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.579939 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:39:06.580301 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:39:06.580309 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 02:39:06.580366 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 02:39:06.580459 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":266,"threads_started":1}
I20260621 02:39:06.580796 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.575276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43231 (local address 127.0.97.3:33635)
0621 02:39:06.575455 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:06.575461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:06.575808 (+ 347us) server_negotiation.cc:407] Connection header received
0621 02:39:06.575997 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:06.576003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:06.576091 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:06.576190 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:06.576549 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.577732 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.578989 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.579637 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.579757 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.580096 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:06.580214 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:06.580302 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:06.580403 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 02:39:06.580484 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:39:06.622720 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.612292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49611 (local address 127.0.97.1:44443)
0621 02:39:06.612446 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:06.612452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:06.617082 (+ 4630us) server_negotiation.cc:407] Connection header received
0621 02:39:06.617264 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:06.617270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:06.617366 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:06.617469 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:06.618098 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.619219 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.620812 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.621261 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.621421 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.622000 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:06.622134 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:06.622222 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:06.622369 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 02:39:06.622450 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 02:39:06.623570 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.611981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44443 (local address 127.0.97.3:49611)
0621 02:39:06.616970 (+ 4989us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:06.616986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:06.617145 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:06.617482 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:06.617487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:06.617498 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:39:06.617918 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.617930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.619415 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.619420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.620673 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:06.620684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.621467 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.621476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:06.621613 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.621670 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:39:06.622222 (+ 552us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:39:06.622236 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 02:39:06.622942 (+ 706us) client_negotiation.cc:240] Negotiation successful
0621 02:39:06.623073 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4792,"thread_start_us":4656,"threads_started":1}
I20260621 02:39:06.666493 7952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:06.666738 7952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:06.669512 7952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d: Sending DeleteTablet for 3 replicas of tablet 8fc63bd5e67446e58825bf0af835a442
I20260621 02:39:06.670171 8089 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc63bd5e67446e58825bf0af835a442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42384
I20260621 02:39:06.670462 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc63bd5e67446e58825bf0af835a442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43674
I20260621 02:39:06.671617 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc63bd5e67446e58825bf0af835a442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48648
I20260621 02:39:06.686075 8532 tablet_replica.cc:330] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: stopping tablet replica
I20260621 02:39:06.686308 8532 raft_consensus.cc:2229] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:06.686589 8532 raft_consensus.cc:2258] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:06.687178 8531 tablet_replica.cc:330] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: stopping tablet replica
I20260621 02:39:06.687371 8531 raft_consensus.cc:2229] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:06.687629 8531 raft_consensus.cc:2258] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:06.687714 8532 ts_tablet_manager.cc:1905] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:06.688158 8530 tablet_replica.cc:330] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: stopping tablet replica
I20260621 02:39:06.688360 8530 raft_consensus.cc:2229] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:39:06.688747 8530 raft_consensus.cc:2258] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:06.689018 8531 ts_tablet_manager.cc:1905] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:06.689736 8530 ts_tablet_manager.cc:1905] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:06.697759 8531 ts_tablet_manager.cc:1918] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:39:06.697966 8531 log.cc:1196] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/8fc63bd5e67446e58825bf0af835a442
I20260621 02:39:06.698475 8531 ts_tablet_manager.cc:1939] T 8fc63bd5e67446e58825bf0af835a442 P 6b1e4a6f17f94606a214371f619cdf14: Deleting consensus metadata
I20260621 02:39:06.698818 8532 ts_tablet_manager.cc:1918] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:39:06.698944 8532 log.cc:1196] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/8fc63bd5e67446e58825bf0af835a442
I20260621 02:39:06.699412 8532 ts_tablet_manager.cc:1939] T 8fc63bd5e67446e58825bf0af835a442 P ed27d65c42df4b0184372b623fe0f63b: Deleting consensus metadata
I20260621 02:39:06.699867 7938 catalog_manager.cc:4841] TS 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885): tablet 8fc63bd5e67446e58825bf0af835a442 (table test-table [id=7737ec3cafb14847bf1a36a9f2b82140]) successfully deleted
I20260621 02:39:06.700470 7939 catalog_manager.cc:4841] TS ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635): tablet 8fc63bd5e67446e58825bf0af835a442 (table test-table [id=7737ec3cafb14847bf1a36a9f2b82140]) successfully deleted
I20260621 02:39:06.704443 8530 ts_tablet_manager.cc:1918] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:39:06.704577 8530 log.cc:1196] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/8fc63bd5e67446e58825bf0af835a442
I20260621 02:39:06.705020 8530 ts_tablet_manager.cc:1939] T 8fc63bd5e67446e58825bf0af835a442 P 13c35b1c0efc4ec391940f02110c7d1e: Deleting consensus metadata
I20260621 02:39:06.706099 7936 catalog_manager.cc:4841] TS 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1:44443): tablet 8fc63bd5e67446e58825bf0af835a442 (table test-table [id=7737ec3cafb14847bf1a36a9f2b82140]) successfully deleted
Jun 21 02:39:06 dist-test-slave-jkn7 krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:39:06 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:06 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:39:06.742479 8540 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:06.731043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.0.97.62:42841)
0621 02:39:06.732001 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:06.732006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:06.732024 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:39:06.732075 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:06.732079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:06.732141 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:06.732266 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:06.733907 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.734627 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.736469 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.736860 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.736947 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.742170 (+ 5223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54590: BlockingRecv error: recv got EOF from 127.0.0.1:54590 (error 108)
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":733,"threads_started":1}
Jun 21 02:39:06 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260621 02:39:06.759737 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.745217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.0.97.62:42841)
0621 02:39:06.745439 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:06.745445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:06.745792 (+ 347us) server_negotiation.cc:407] Connection header received
0621 02:39:06.745967 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:06.745972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:06.746081 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:06.746183 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:06.747384 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.748446 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.749767 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.750407 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.750540 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.754039 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:06.754053 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:06.754057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:06.754095 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:06.757247 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:06.757885 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:06.757890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:06.757893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:06.758036 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:06.758394 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:06.758399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:06.758402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:06.758726 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:06.758977 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:06.759188 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 02:39:06.759402 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260621 02:39:06.763504 7952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:39:06.764168 7952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:39:06.770613 8089 tablet_service.cc:1467] Processing CreateTablet for tablet dbfecacef4724249af5a2102a271e24e (DEFAULT_TABLE table=test-table [id=f524b5f08151499fa0bade8e2ca6c938]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:06.771092 8089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfecacef4724249af5a2102a271e24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:06.771041 8232 tablet_service.cc:1467] Processing CreateTablet for tablet dbfecacef4724249af5a2102a271e24e (DEFAULT_TABLE table=test-table [id=f524b5f08151499fa0bade8e2ca6c938]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:06.771430 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfecacef4724249af5a2102a271e24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:06.776010 8546 tablet_bootstrap.cc:492] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: Bootstrap starting.
I20260621 02:39:06.776041 8367 tablet_service.cc:1467] Processing CreateTablet for tablet dbfecacef4724249af5a2102a271e24e (DEFAULT_TABLE table=test-table [id=f524b5f08151499fa0bade8e2ca6c938]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:06.776540 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfecacef4724249af5a2102a271e24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:06.777964 8546 tablet_bootstrap.cc:654] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:06.786259 8546 tablet_bootstrap.cc:492] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: No bootstrap required, opened a new log
I20260621 02:39:06.786505 8546 ts_tablet_manager.cc:1397] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260621 02:39:06.786840 8549 tablet_bootstrap.cc:492] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: Bootstrap starting.
I20260621 02:39:06.787592 8546 raft_consensus.cc:348] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.787850 8547 tablet_bootstrap.cc:492] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: Bootstrap starting.
I20260621 02:39:06.787964 8546 raft_consensus.cc:374] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:06.788125 8546 raft_consensus.cc:729] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1e4a6f17f94606a214371f619cdf14, State: Initialized, Role: FOLLOWER
I20260621 02:39:06.788496 8546 consensus_queue.cc:260] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.788812 8549 tablet_bootstrap.cc:654] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:06.789199 8546 ts_tablet_manager.cc:1428] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 02:39:06.790863 8547 tablet_bootstrap.cc:654] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:06.793856 8547 tablet_bootstrap.cc:492] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: No bootstrap required, opened a new log
I20260621 02:39:06.793856 8549 tablet_bootstrap.cc:492] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: No bootstrap required, opened a new log
I20260621 02:39:06.794004 8547 ts_tablet_manager.cc:1397] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 02:39:06.794080 8549 ts_tablet_manager.cc:1397] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260621 02:39:06.794771 8547 raft_consensus.cc:348] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.794826 8549 raft_consensus.cc:348] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.795023 8547 raft_consensus.cc:374] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:06.795051 8549 raft_consensus.cc:374] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:06.795127 8547 raft_consensus.cc:729] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c35b1c0efc4ec391940f02110c7d1e, State: Initialized, Role: FOLLOWER
I20260621 02:39:06.795166 8549 raft_consensus.cc:729] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed27d65c42df4b0184372b623fe0f63b, State: Initialized, Role: FOLLOWER
I20260621 02:39:06.795408 8547 consensus_queue.cc:260] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.795413 8549 consensus_queue.cc:260] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.795969 8549 ts_tablet_manager.cc:1428] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:39:06.795969 8547 ts_tablet_manager.cc:1428] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 02:39:06.935150 8515 raft_consensus.cc:482] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:39:06.935395 8515 raft_consensus.cc:504] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.936208 8515 leader_election.cc:288] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885), ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635)
I20260621 02:39:06.936776 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfecacef4724249af5a2102a271e24e" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed27d65c42df4b0184372b623fe0f63b" is_pre_election: true
I20260621 02:39:06.936793 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfecacef4724249af5a2102a271e24e" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e4a6f17f94606a214371f619cdf14" is_pre_election: true
I20260621 02:39:06.937047 8392 raft_consensus.cc:2454] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 0.
I20260621 02:39:06.937063 8252 raft_consensus.cc:2454] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 0.
I20260621 02:39:06.937549 8041 leader_election.cc:302] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c35b1c0efc4ec391940f02110c7d1e, 6b1e4a6f17f94606a214371f619cdf14; no voters:
I20260621 02:39:06.937856 8515 raft_consensus.cc:2781] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:39:06.937994 8515 raft_consensus.cc:482] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:39:06.938107 8515 raft_consensus.cc:3037] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:06.940176 8515 raft_consensus.cc:504] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.940757 8515 leader_election.cc:288] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 election: Requested vote from peers 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885), ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635)
I20260621 02:39:06.941339 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfecacef4724249af5a2102a271e24e" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e4a6f17f94606a214371f619cdf14"
I20260621 02:39:06.941416 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfecacef4724249af5a2102a271e24e" candidate_uuid: "13c35b1c0efc4ec391940f02110c7d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed27d65c42df4b0184372b623fe0f63b"
I20260621 02:39:06.941557 8252 raft_consensus.cc:3037] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:06.941670 8392 raft_consensus.cc:3037] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:06.944003 8392 raft_consensus.cc:2454] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 1.
I20260621 02:39:06.944468 8042 leader_election.cc:302] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c35b1c0efc4ec391940f02110c7d1e, ed27d65c42df4b0184372b623fe0f63b; no voters:
I20260621 02:39:06.944490 8252 raft_consensus.cc:2454] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c35b1c0efc4ec391940f02110c7d1e in term 1.
I20260621 02:39:06.944788 8515 raft_consensus.cc:2781] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:06.944950 8515 raft_consensus.cc:686] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 1 LEADER]: Becoming Leader. State: Replica: 13c35b1c0efc4ec391940f02110c7d1e, State: Running, Role: LEADER
I20260621 02:39:06.945220 8515 consensus_queue.cc:237] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:06.947772 7952 catalog_manager.cc:5495] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "13c35b1c0efc4ec391940f02110c7d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } health_report { overall_health: UNKNOWN } } }
I20260621 02:39:06.993098 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:06.987550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.0.97.1:44443)
0621 02:39:06.987707 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:06.987712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:06.987789 (+ 77us) server_negotiation.cc:407] Connection header received
0621 02:39:06.987975 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:06.987981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:06.988062 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:06.988159 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:06.988523 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.989699 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.991264 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:06.991895 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:06.992036 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:06.992435 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:06.992562 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:06.992651 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:06.992808 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 02:39:06.992876 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 02:39:06.995666 8392 raft_consensus.cc:1261] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Refusing update from remote peer 13c35b1c0efc4ec391940f02110c7d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:06.995688 8252 raft_consensus.cc:1261] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Refusing update from remote peer 13c35b1c0efc4ec391940f02110c7d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:06.996376 8480 consensus_queue.cc:1035] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:06.996634 8515 consensus_queue.cc:1035] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:07.006883 7951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:07.007133 7951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:07.009603 7951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d: Sending DeleteTablet for 3 replicas of tablet dbfecacef4724249af5a2102a271e24e
I20260621 02:39:07.010301 8089 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfecacef4724249af5a2102a271e24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42384
I20260621 02:39:07.010346 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfecacef4724249af5a2102a271e24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43674
I20260621 02:39:07.010561 8530 tablet_replica.cc:330] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: stopping tablet replica
I20260621 02:39:07.010582 8531 tablet_replica.cc:330] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: stopping tablet replica
I20260621 02:39:07.010638 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfecacef4724249af5a2102a271e24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48648
I20260621 02:39:07.010725 8531 raft_consensus.cc:2229] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:07.010761 8530 raft_consensus.cc:2229] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:39:07.010893 8532 tablet_replica.cc:330] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: stopping tablet replica
I20260621 02:39:07.011039 8530 raft_consensus.cc:2258] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:07.011091 8532 raft_consensus.cc:2229] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:07.011113 8531 raft_consensus.cc:2258] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:07.011281 8532 raft_consensus.cc:2258] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:07.012012 8531 ts_tablet_manager.cc:1905] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:07.012020 7951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 02:39:07.012291 8530 ts_tablet_manager.cc:1905] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 02:39:07.012619 7951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:39:07.017879 8532 ts_tablet_manager.cc:1905] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:07.018532 8530 ts_tablet_manager.cc:1918] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:07.018520 8531 ts_tablet_manager.cc:1918] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:07.018667 8531 log.cc:1196] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/dbfecacef4724249af5a2102a271e24e
I20260621 02:39:07.018667 8530 log.cc:1196] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/dbfecacef4724249af5a2102a271e24e
I20260621 02:39:07.019066 8530 ts_tablet_manager.cc:1939] T dbfecacef4724249af5a2102a271e24e P 13c35b1c0efc4ec391940f02110c7d1e: Deleting consensus metadata
I20260621 02:39:07.019097 8531 ts_tablet_manager.cc:1939] T dbfecacef4724249af5a2102a271e24e P 6b1e4a6f17f94606a214371f619cdf14: Deleting consensus metadata
I20260621 02:39:07.019460 8232 tablet_service.cc:1467] Processing CreateTablet for tablet 068778754c75443c898558b7b47a0480 (DEFAULT_TABLE table=test-table [id=d0efa084ee724310a414d58b6f1bc73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:07.019840 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068778754c75443c898558b7b47a0480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:07.020051 7938 catalog_manager.cc:4841] TS 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885): tablet dbfecacef4724249af5a2102a271e24e (table test-table [id=f524b5f08151499fa0bade8e2ca6c938]) successfully deleted
I20260621 02:39:07.022080 8367 tablet_service.cc:1467] Processing CreateTablet for tablet 068778754c75443c898558b7b47a0480 (DEFAULT_TABLE table=test-table [id=d0efa084ee724310a414d58b6f1bc73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:07.022387 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068778754c75443c898558b7b47a0480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:07.024659 8532 ts_tablet_manager.cc:1918] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:07.024816 8532 log.cc:1196] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/dbfecacef4724249af5a2102a271e24e
I20260621 02:39:07.025252 8532 ts_tablet_manager.cc:1939] T dbfecacef4724249af5a2102a271e24e P ed27d65c42df4b0184372b623fe0f63b: Deleting consensus metadata
I20260621 02:39:07.025965 7936 catalog_manager.cc:4841] TS 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1:44443): tablet dbfecacef4724249af5a2102a271e24e (table test-table [id=f524b5f08151499fa0bade8e2ca6c938]) successfully deleted
I20260621 02:39:07.026429 8089 tablet_service.cc:1467] Processing CreateTablet for tablet 068778754c75443c898558b7b47a0480 (DEFAULT_TABLE table=test-table [id=d0efa084ee724310a414d58b6f1bc73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:07.026830 8089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068778754c75443c898558b7b47a0480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:07.027485 8546 tablet_bootstrap.cc:492] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: Bootstrap starting.
I20260621 02:39:07.027513 7939 catalog_manager.cc:4841] TS ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635): tablet dbfecacef4724249af5a2102a271e24e (table test-table [id=f524b5f08151499fa0bade8e2ca6c938]) successfully deleted
I20260621 02:39:07.027860 8549 tablet_bootstrap.cc:492] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: Bootstrap starting.
I20260621 02:39:07.029153 8546 tablet_bootstrap.cc:654] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:07.029482 8549 tablet_bootstrap.cc:654] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:07.031869 8546 tablet_bootstrap.cc:492] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: No bootstrap required, opened a new log
I20260621 02:39:07.032042 8546 ts_tablet_manager.cc:1397] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260621 02:39:07.032234 8547 tablet_bootstrap.cc:492] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: Bootstrap starting.
I20260621 02:39:07.032828 8546 raft_consensus.cc:348] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:07.033002 8546 raft_consensus.cc:374] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:07.033053 8546 raft_consensus.cc:729] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1e4a6f17f94606a214371f619cdf14, State: Initialized, Role: FOLLOWER
I20260621 02:39:07.033578 8546 consensus_queue.cc:260] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:07.034533 8547 tablet_bootstrap.cc:654] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:07.034600 8546 ts_tablet_manager.cc:1428] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 02:39:07.036481 8547 tablet_bootstrap.cc:492] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: No bootstrap required, opened a new log
I20260621 02:39:07.036660 8547 ts_tablet_manager.cc:1397] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 02:39:07.037417 8547 raft_consensus.cc:348] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:07.037628 8547 raft_consensus.cc:374] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:07.037729 8547 raft_consensus.cc:729] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c35b1c0efc4ec391940f02110c7d1e, State: Initialized, Role: FOLLOWER
I20260621 02:39:07.038022 8547 consensus_queue.cc:260] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:07.039337 8547 ts_tablet_manager.cc:1428] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 02:39:07.042164 8549 tablet_bootstrap.cc:492] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: No bootstrap required, opened a new log
I20260621 02:39:07.042346 8549 ts_tablet_manager.cc:1397] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260621 02:39:07.043076 8549 raft_consensus.cc:348] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:07.043298 8549 raft_consensus.cc:374] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:07.043397 8549 raft_consensus.cc:729] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed27d65c42df4b0184372b623fe0f63b, State: Initialized, Role: FOLLOWER
I20260621 02:39:07.043628 8549 consensus_queue.cc:260] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:07.044200 8549 ts_tablet_manager.cc:1428] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 02:39:07.093592 8517 raft_consensus.cc:482] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:39:07.093848 8517 raft_consensus.cc:504] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:07.095286 8517 leader_election.cc:288] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1:44443), ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635)
I20260621 02:39:07.101217 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.095657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41969 (local address 127.0.97.3:33635)
0621 02:39:07.096551 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:07.096557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:07.096578 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:39:07.096632 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:07.096636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:07.096687 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:07.096789 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:07.097230 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.097962 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.099530 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.100312 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.100458 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.100568 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:07.100801 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 02:39:07.100931 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":668,"threads_started":1}
I20260621 02:39:07.101251 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.095508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33635 (local address 127.0.97.2:41969)
0621 02:39:07.096279 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:07.096303 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:07.096432 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:07.096808 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:07.096813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:07.096839 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:07.097091 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:07.097100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.098091 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.098097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:07.099390 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:07.099401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.100473 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.100479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:07.100656 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.100678 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:39:07.100777 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 02:39:07.100913 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":1,"thread_start_us":553,"threads_started":1}
I20260621 02:39:07.101626 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.095508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44443 (local address 127.0.97.2:56169)
0621 02:39:07.096803 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:07.096821 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:07.096965 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:07.097271 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:07.097275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:07.097319 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:07.097606 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:07.097613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.098506 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.098517 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:07.099798 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:07.099809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.100827 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.100833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:07.101003 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.101025 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:39:07.101120 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:39:07.101245 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"mutex_wait_us":38,"thread_start_us":1018,"threads_started":1}
I20260621 02:39:07.101815 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068778754c75443c898558b7b47a0480" candidate_uuid: "6b1e4a6f17f94606a214371f619cdf14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed27d65c42df4b0184372b623fe0f63b" is_pre_election: true
I20260621 02:39:07.101917 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.095666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56169 (local address 127.0.97.1:44443)
0621 02:39:07.095900 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:07.095910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:07.096902 (+ 992us) server_negotiation.cc:407] Connection header received
0621 02:39:07.097096 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:07.097102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:07.097168 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:07.097260 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:07.097749 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.098384 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.099930 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.100648 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.100787 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.100890 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:07.101150 (+ 260us) server_negotiation.cc:299] Negotiation successful
0621 02:39:07.101631 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 02:39:07.102049 8392 raft_consensus.cc:2454] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1e4a6f17f94606a214371f619cdf14 in term 0.
I20260621 02:39:07.102422 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068778754c75443c898558b7b47a0480" candidate_uuid: "6b1e4a6f17f94606a214371f619cdf14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c35b1c0efc4ec391940f02110c7d1e" is_pre_election: true
I20260621 02:39:07.102459 8182 leader_election.cc:302] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1e4a6f17f94606a214371f619cdf14, ed27d65c42df4b0184372b623fe0f63b; no voters:
I20260621 02:39:07.102636 8114 raft_consensus.cc:2454] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1e4a6f17f94606a214371f619cdf14 in term 0.
I20260621 02:39:07.102813 8517 raft_consensus.cc:2781] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:39:07.102943 8517 raft_consensus.cc:482] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:39:07.103083 8517 raft_consensus.cc:3037] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:07.105156 8517 raft_consensus.cc:504] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:07.105717 8517 leader_election.cc:288] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [CANDIDATE]: Term 1 election: Requested vote from peers 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1:44443), ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635)
I20260621 02:39:07.106069 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068778754c75443c898558b7b47a0480" candidate_uuid: "6b1e4a6f17f94606a214371f619cdf14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c35b1c0efc4ec391940f02110c7d1e"
I20260621 02:39:07.106158 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068778754c75443c898558b7b47a0480" candidate_uuid: "6b1e4a6f17f94606a214371f619cdf14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed27d65c42df4b0184372b623fe0f63b"
I20260621 02:39:07.106256 8114 raft_consensus.cc:3037] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:07.106383 8392 raft_consensus.cc:3037] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:07.108562 8392 raft_consensus.cc:2454] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1e4a6f17f94606a214371f619cdf14 in term 1.
I20260621 02:39:07.108577 8114 raft_consensus.cc:2454] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1e4a6f17f94606a214371f619cdf14 in term 1.
I20260621 02:39:07.108909 8182 leader_election.cc:302] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1e4a6f17f94606a214371f619cdf14, ed27d65c42df4b0184372b623fe0f63b; no voters:
I20260621 02:39:07.109191 8517 raft_consensus.cc:2781] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:07.109583 8517 raft_consensus.cc:686] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 LEADER]: Becoming Leader. State: Replica: 6b1e4a6f17f94606a214371f619cdf14, State: Running, Role: LEADER
I20260621 02:39:07.109926 8517 consensus_queue.cc:237] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } }
I20260621 02:39:07.112869 7950 catalog_manager.cc:5495] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6b1e4a6f17f94606a214371f619cdf14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1e4a6f17f94606a214371f619cdf14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 } health_report { overall_health: UNKNOWN } } }
I20260621 02:39:07.144912 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.138209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.0.97.2:35885)
0621 02:39:07.138961 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:07.138965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:07.138982 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:39:07.139051 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:07.139054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:07.139120 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:07.139251 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:07.139725 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.140494 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.142390 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.142917 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.143038 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.143556 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:07.144245 (+ 689us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:07.144360 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:07.144479 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 02:39:07.144554 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":535,"threads_started":1}
I20260621 02:39:07.158957 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.154321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45617 (local address 127.0.97.3:33635)
0621 02:39:07.154482 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:07.154487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:07.154878 (+ 391us) server_negotiation.cc:407] Connection header received
0621 02:39:07.155011 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:07.155014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:07.155075 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:07.155158 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:07.155546 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.156193 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.157580 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.157969 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.158061 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.158446 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:07.158540 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:07.158603 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:07.158715 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 02:39:07.158769 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:39:07.159045 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.154243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33635 (local address 127.0.97.2:45617)
0621 02:39:07.154794 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:07.154809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:07.154914 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:07.155181 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:07.155184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:07.155193 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:39:07.155433 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:07.155439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.156318 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.156323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:07.157470 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:07.157481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.158136 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.158141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:07.158255 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.158300 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:39:07.158615 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:39:07.158622 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:39:07.158681 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 02:39:07.158769 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":272,"threads_started":1}
I20260621 02:39:07.166261 8392 raft_consensus.cc:1261] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Refusing update from remote peer 6b1e4a6f17f94606a214371f619cdf14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:07.166306 8113 raft_consensus.cc:1261] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Refusing update from remote peer 6b1e4a6f17f94606a214371f619cdf14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:07.166917 8562 consensus_queue.cc:1035] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed27d65c42df4b0184372b623fe0f63b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:07.167240 8517 consensus_queue.cc:1035] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c35b1c0efc4ec391940f02110c7d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:07.190858 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.185873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46995 (local address 127.0.97.2:35885)
0621 02:39:07.186021 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:07.186025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:07.186530 (+ 505us) server_negotiation.cc:407] Connection header received
0621 02:39:07.186689 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:07.186694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:07.186767 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:07.186837 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:07.187236 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.187926 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.189307 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.189878 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.190004 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.190332 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:07.190422 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:07.190489 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:07.190577 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 02:39:07.190643 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 02:39:07.190908 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.185866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35885 (local address 127.0.97.3:46995)
0621 02:39:07.186428 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:07.186449 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:07.186576 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:07.186860 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:07.186863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:07.186872 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:39:07.187078 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:07.187086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.188063 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.188068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:07.189147 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:07.189160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.190028 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.190034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:07.190186 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.190222 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:39:07.190500 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:39:07.190508 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 02:39:07.190565 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 02:39:07.190663 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":280,"threads_started":1}
I20260621 02:39:07.221449 7950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:07.221701 7950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:07.224568 7950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c15e384d28374935972cb547ccd3eb9d: Sending DeleteTablet for 3 replicas of tablet 068778754c75443c898558b7b47a0480
I20260621 02:39:07.225425 8089 tablet_service.cc:1514] Processing DeleteTablet for tablet 068778754c75443c898558b7b47a0480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42384
I20260621 02:39:07.225438 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet 068778754c75443c898558b7b47a0480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43674
I20260621 02:39:07.225728 8530 tablet_replica.cc:330] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: stopping tablet replica
I20260621 02:39:07.225745 8531 tablet_replica.cc:330] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: stopping tablet replica
I20260621 02:39:07.225916 8531 raft_consensus.cc:2229] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:39:07.225932 8530 raft_consensus.cc:2229] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:07.226141 8530 raft_consensus.cc:2258] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:07.226306 8531 raft_consensus.cc:2258] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:07.226516 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet 068778754c75443c898558b7b47a0480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48648
I20260621 02:39:07.226804 8532 tablet_replica.cc:330] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: stopping tablet replica
I20260621 02:39:07.226982 8532 raft_consensus.cc:2229] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:07.227180 8532 raft_consensus.cc:2258] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:07.227672 8530 ts_tablet_manager.cc:1905] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:07.227780 8531 ts_tablet_manager.cc:1905] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:07.232637 8531 ts_tablet_manager.cc:1918] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:39:07.232790 8531 log.cc:1196] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/068778754c75443c898558b7b47a0480
I20260621 02:39:07.233218 8531 ts_tablet_manager.cc:1939] T 068778754c75443c898558b7b47a0480 P 6b1e4a6f17f94606a214371f619cdf14: Deleting consensus metadata
I20260621 02:39:07.234223 8530 ts_tablet_manager.cc:1918] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:39:07.234351 8530 log.cc:1196] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/068778754c75443c898558b7b47a0480
I20260621 02:39:07.234627 7938 catalog_manager.cc:4841] TS 6b1e4a6f17f94606a214371f619cdf14 (127.0.97.2:35885): tablet 068778754c75443c898558b7b47a0480 (table test-table [id=d0efa084ee724310a414d58b6f1bc73b]) successfully deleted
I20260621 02:39:07.234714 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 8023
I20260621 02:39:07.234762 8530 ts_tablet_manager.cc:1939] T 068778754c75443c898558b7b47a0480 P 13c35b1c0efc4ec391940f02110c7d1e: Deleting consensus metadata
I20260621 02:39:07.235707 7936 catalog_manager.cc:4841] TS 13c35b1c0efc4ec391940f02110c7d1e (127.0.97.1:44443): tablet 068778754c75443c898558b7b47a0480 (table test-table [id=d0efa084ee724310a414d58b6f1bc73b]) successfully deleted
I20260621 02:39:07.237118 8532 ts_tablet_manager.cc:1905] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:07.242020 8532 ts_tablet_manager.cc:1918] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:39:07.242169 8532 log.cc:1196] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/068778754c75443c898558b7b47a0480
I20260621 02:39:07.242597 8532 ts_tablet_manager.cc:1939] T 068778754c75443c898558b7b47a0480 P ed27d65c42df4b0184372b623fe0f63b: Deleting consensus metadata
I20260621 02:39:07.243649 7939 catalog_manager.cc:4841] TS ed27d65c42df4b0184372b623fe0f63b (127.0.97.3:33635): tablet 068778754c75443c898558b7b47a0480 (table test-table [id=d0efa084ee724310a414d58b6f1bc73b]) successfully deleted
Jun 21 02:39:07 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260621 02:39:07.253508 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.236019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.0.97.1:44443)
0621 02:39:07.236190 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:07.236196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:07.237107 (+ 911us) server_negotiation.cc:407] Connection header received
0621 02:39:07.237330 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:07.237336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:07.237422 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:07.237495 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:07.239174 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.240201 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.242074 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.242764 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.242914 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.247648 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:07.247662 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:07.247677 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:07.247712 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:07.250761 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:07.251446 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:07.251453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:07.251455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:07.251572 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:07.251926 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:07.251933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:07.251936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:07.252278 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:07.252598 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:07.252876 (+ 278us) server_negotiation.cc:299] Negotiation successful
0621 02:39:07.253123 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:39:07.254191 8154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:07.380262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 8023
I20260621 02:39:07.413237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 8163
Jun 21 02:39:07 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260621 02:39:07.433756 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.414038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.0.97.2:35885)
0621 02:39:07.414248 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:07.414254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:07.414275 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:39:07.414428 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:07.414433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:07.414507 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:07.414620 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:07.415785 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.416964 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.418449 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.419111 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.419296 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.427072 (+ 7776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:07.427086 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:07.427107 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:07.427162 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:07.431090 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:07.431783 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:07.431791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:07.431795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:07.431938 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:07.432257 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:07.432261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:07.432265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:07.432678 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:07.432976 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:07.433140 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 02:39:07.433390 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:39:07.434518 8293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:07.555312 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 8163
I20260621 02:39:07.575347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 8302
Jun 21 02:39:07 dist-test-slave-jkn7 krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260621 02:39:07.590071 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:07.576094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.0.97.3:33635)
0621 02:39:07.576273 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:07.576281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:07.576357 (+ 76us) server_negotiation.cc:407] Connection header received
0621 02:39:07.576584 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:07.576588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:07.576648 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:07.576750 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:07.578084 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.578826 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.580039 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:07.580428 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:07.580564 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:07.584886 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:07.584899 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:07.584916 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:07.584951 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:07.587571 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:07.588212 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:07.588218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:07.588221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:07.588329 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:07.588611 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:07.588614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:07.588616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:07.588947 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:07.589151 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:07.589661 (+ 510us) server_negotiation.cc:299] Negotiation successful
0621 02:39:07.589808 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:39:07.590747 8432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:07.716926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 8302
I20260621 02:39:07.738293 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 7920
I20260621 02:39:07.739413 7992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:07.852101 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 7920
2026-06-21T02:39:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5604 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:39:07 dist-test-slave-jkn7 krb5kdc[8584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:39:07 dist-test-slave-jkn7 krb5kdc[8584](info): set up 2 sockets
Jun 21 02:39:07 dist-test-slave-jkn7 krb5kdc[8584](info): commencing operation
krb5kdc: starting...
W20260621 02:39:08.946365 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:39:09 dist-test-slave-jkn7 krb5kdc[8584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:39:09.161962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39273
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:09.474896 8597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:09.475153 8597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:09.475317 8597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:09.482256 8597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:39:09.482353 8597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:09.482450 8597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:09.482568 8597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:39:09.482631 8597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:39:09.491842 8597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:09.493021 8597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:09.494275 8597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:09.505473 8605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:09.505426 8602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:09.505501 8603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:09.506209 8597 server_base.cc:1034] running on GCE node
I20260621 02:39:09.506974 8597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:09.508343 8597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:09.509557 8597 hybrid_clock.cc:648] HybridClock initialized: now 1782009549509500 us; error 46 us; skew 500 ppm
I20260621 02:39:09.509915 8597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T02:39:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1724 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:39:09 dist-test-slave-jkn7 krb5kdc[8611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:39:09 dist-test-slave-jkn7 krb5kdc[8611](info): set up 2 sockets
Jun 21 02:39:09 dist-test-slave-jkn7 krb5kdc[8611](info): commencing operation
krb5kdc: starting...
W20260621 02:39:10.670256 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:39:10 dist-test-slave-jkn7 krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:39:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:39:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:39:10.886874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32833
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:11.201295 8624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:11.201586 8624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:11.201683 8624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:11.208521 8624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:39:11.208613 8624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:11.208659 8624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:11.208719 8624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:39:11.208762 8624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:39:11.217942 8624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:11.219084 8624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:11.220304 8624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:11.231900 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:11.231968 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:11.231770 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:11.232966 8624 server_base.cc:1034] running on GCE node
I20260621 02:39:11.233783 8624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:11.235126 8624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:11.236342 8624 hybrid_clock.cc:648] HybridClock initialized: now 1782009551236303 us; error 27 us; skew 500 ppm
I20260621 02:39:11.236642 8624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:11 dist-test-slave-jkn7 krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:11.239830 8624 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:39:11.242115 8624 webserver.cc:460] Webserver started at http://127.0.97.62:46053/ using document root <none> and password file <none>
I20260621 02:39:11.242758 8624 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:11.242854 8624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:11.243091 8624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:11.245947 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "aab74b9b7f1b407b93b68456c60bc0c0"
format_stamp: "Formatted at 2026-06-21 02:39:11 on dist-test-slave-jkn7"
server_key: "91166babb7dff7151b2b22604c34c475"
server_key_iv: "16db0b3526798c531c81876441919663"
server_key_version: "encryptionkey@0"
I20260621 02:39:11.246649 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "aab74b9b7f1b407b93b68456c60bc0c0"
format_stamp: "Formatted at 2026-06-21 02:39:11 on dist-test-slave-jkn7"
server_key: "91166babb7dff7151b2b22604c34c475"
server_key_iv: "16db0b3526798c531c81876441919663"
server_key_version: "encryptionkey@0"
I20260621 02:39:11.251421 8624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:39:11.254747 8639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:11.256016 8624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:39:11.256184 8624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "aab74b9b7f1b407b93b68456c60bc0c0"
format_stamp: "Formatted at 2026-06-21 02:39:11 on dist-test-slave-jkn7"
server_key: "91166babb7dff7151b2b22604c34c475"
server_key_iv: "16db0b3526798c531c81876441919663"
server_key_version: "encryptionkey@0"
I20260621 02:39:11.256366 8624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:11.282492 8624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:11.286931 8624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:11.287246 8624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:11.304891 8624 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32833
I20260621 02:39:11.304929 8690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32833 every 8 connection(s)
I20260621 02:39:11.306336 8624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:39:11.310271 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:11.315717 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 8624
I20260621 02:39:11.315922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:39:11.316537 388 external_mini_cluster.cc:1428] Setting key bb3c41819df5dd3f3101084a661eee5f
I20260621 02:39:11.321913 8691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: Bootstrap starting.
I20260621 02:39:11.325583 8691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:11.326735 8691 log.cc:824] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:11.329686 8691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: No bootstrap required, opened a new log
Jun 21 02:39:11 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:39:11.339568 8691 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } }
I20260621 02:39:11.340013 8691 raft_consensus.cc:374] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:11.340178 8691 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab74b9b7f1b407b93b68456c60bc0c0, State: Initialized, Role: FOLLOWER
I20260621 02:39:11.340935 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:11.318563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.97.62:32833)
0621 02:39:11.319395 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:11.319411 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:11.319469 (+ 58us) server_negotiation.cc:407] Connection header received
0621 02:39:11.321812 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:11.321839 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:11.322332 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:11.322754 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:11.323980 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:11.326038 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:11.327515 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:11.328367 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:11.328538 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:11.332409 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:11.332420 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:11.332444 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:11.332486 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:11.336476 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:11.337137 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:11.337150 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:11.337158 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:11.337396 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:11.337793 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:11.337798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:11.337801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:11.338505 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:11.338794 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:11.339092 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 02:39:11.339506 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":313,"threads_started":1}
I20260621 02:39:11.340842 8691 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } }
I20260621 02:39:11.341125 8691 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:39:11.341323 8691 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:39:11.341572 8691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:11.343910 8691 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } }
I20260621 02:39:11.344385 8691 leader_election.cc:302] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab74b9b7f1b407b93b68456c60bc0c0; no voters:
I20260621 02:39:11.344964 8691 leader_election.cc:288] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:39:11.345082 8696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:11.345726 8696 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 1 LEADER]: Becoming Leader. State: Replica: aab74b9b7f1b407b93b68456c60bc0c0, State: Running, Role: LEADER
I20260621 02:39:11.346300 8696 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } }
I20260621 02:39:11.347702 8691 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:39:11.349162 8698 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } } }
I20260621 02:39:11.349593 8698 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:11.349496 8697 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab74b9b7f1b407b93b68456c60bc0c0. Latest consensus state: current_term: 1 leader_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } } }
I20260621 02:39:11.349792 8697 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:11.352710 8702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:39:11.357942 8702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:39:11.373508 8702 catalog_manager.cc:1347] Generated new cluster ID: 9cdbc08b8159428a99a7f4e4285940bf
I20260621 02:39:11.373659 8702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:39:11.390592 8702 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:39:11.391860 8702 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:39:11.402714 8702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: Generated new TSK 0
I20260621 02:39:11.403497 8702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:39:11.456619 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32833
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:11.784950 8717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:11.785199 8717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:11.785339 8717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:11.792300 8717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:11.792397 8717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:11.792502 8717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:39:11.801797 8717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:11.802942 8717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:11.804200 8717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:11.817858 8725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:11.817929 8723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:11.817890 8722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:11.819092 8717 server_base.cc:1034] running on GCE node
I20260621 02:39:11.820050 8717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:11.821067 8717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:11.822327 8717 hybrid_clock.cc:648] HybridClock initialized: now 1782009551822288 us; error 53 us; skew 500 ppm
I20260621 02:39:11.822723 8717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:11 dist-test-slave-jkn7 krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:11.825798 8717 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:39:11.827621 8717 webserver.cc:460] Webserver started at http://127.0.97.1:36345/ using document root <none> and password file <none>
I20260621 02:39:11.828549 8717 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:11.828740 8717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:11.829106 8717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:11.832347 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "d69996eb256d464781cd10c5a59d8f11"
format_stamp: "Formatted at 2026-06-21 02:39:11 on dist-test-slave-jkn7"
server_key: "73b94419b5cb14d82fb933e7e5efc230"
server_key_iv: "227b2d554cee78fc6ac58aee71d3c65d"
server_key_version: "encryptionkey@0"
I20260621 02:39:11.833099 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d69996eb256d464781cd10c5a59d8f11"
format_stamp: "Formatted at 2026-06-21 02:39:11 on dist-test-slave-jkn7"
server_key: "73b94419b5cb14d82fb933e7e5efc230"
server_key_iv: "227b2d554cee78fc6ac58aee71d3c65d"
server_key_version: "encryptionkey@0"
I20260621 02:39:11.838387 8717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260621 02:39:11.841702 8732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:11.843070 8717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:39:11.843230 8717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "d69996eb256d464781cd10c5a59d8f11"
format_stamp: "Formatted at 2026-06-21 02:39:11 on dist-test-slave-jkn7"
server_key: "73b94419b5cb14d82fb933e7e5efc230"
server_key_iv: "227b2d554cee78fc6ac58aee71d3c65d"
server_key_version: "encryptionkey@0"
I20260621 02:39:11.843403 8717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:11.871455 8717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:11.875483 8717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:11.875780 8717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:11.876739 8717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:11.878114 8717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:11.878185 8717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:11.878346 8717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:11.878410 8717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:11.911448 8717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40383
I20260621 02:39:11.911495 8844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40383 every 8 connection(s)
I20260621 02:39:11.912767 8717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:39:11.918965 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 8717
I20260621 02:39:11.919225 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:39:11.920045 388 external_mini_cluster.cc:1428] Setting key 59936e339fe13ef2059319cdcfc5e81a
Jun 21 02:39:11 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:39:11.933386 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:11.915467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54481 (local address 127.0.97.62:32833)
0621 02:39:11.915913 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:11.915917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:11.916878 (+ 961us) server_negotiation.cc:407] Connection header received
0621 02:39:11.918845 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:11.918850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:11.918914 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:11.919008 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:11.920980 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:11.921926 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:11.923564 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:11.923961 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:11.924091 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:11.927836 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:11.927843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:11.927853 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:11.927886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:11.930386 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:11.931118 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:11.931123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:11.931127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:11.931254 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:11.931669 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:11.931676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:11.931682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:11.931936 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:11.932087 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:11.932762 (+ 675us) server_negotiation.cc:299] Negotiation successful
0621 02:39:11.932981 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":217,"threads_started":1}
I20260621 02:39:11.934394 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:11.916062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:54481)
0621 02:39:11.916716 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:11.916758 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:11.918511 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:11.919294 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:11.919307 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:11.920076 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:11.920820 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:11.920840 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:11.922065 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:11.922071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:11.923420 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:11.923431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:11.924192 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:11.924201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:11.924435 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:11.925163 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:11.925187 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:11.927613 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:11.930531 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:11.930540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:11.930557 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:11.930986 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:11.931365 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:11.931370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:11.931374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:11.931558 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:11.932062 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:11.932070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:11.932405 (+ 335us) client_negotiation.cc:769] Sending connection context
0621 02:39:11.932677 (+ 272us) client_negotiation.cc:240] Negotiation successful
0621 02:39:11.933208 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":255,"threads_started":1}
I20260621 02:39:11.935997 8845 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32833
I20260621 02:39:11.936314 8845 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:11.937022 8845 heartbeater.cc:511] Master 127.0.97.62:32833 requested a full tablet report, sending...
I20260621 02:39:11.939422 8656 ts_manager.cc:194] Registered new tserver with Master: d69996eb256d464781cd10c5a59d8f11 (127.0.97.1:40383)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260621 02:39:11.941689 8656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:39:11.968788 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32833
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:12.282819 8851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:12.283094 8851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:12.283267 8851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:12.290512 8851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:12.290621 8851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:12.290797 8851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:39:12.300179 8851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:12.301478 8851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:12.302727 8851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:12.315675 8857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:12.315718 8859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:12.316300 8856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:12.316560 8851 server_base.cc:1034] running on GCE node
I20260621 02:39:12.317215 8851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:12.318208 8851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:12.319451 8851 hybrid_clock.cc:648] HybridClock initialized: now 1782009552319391 us; error 45 us; skew 500 ppm
I20260621 02:39:12.319802 8851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:12 dist-test-slave-jkn7 krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:12.323141 8851 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:39:12.324843 8851 webserver.cc:460] Webserver started at http://127.0.97.2:35681/ using document root <none> and password file <none>
I20260621 02:39:12.325693 8851 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:12.325850 8851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:12.326200 8851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:12.329603 8851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "d73f7cdefde44f67984af4849852f28a"
format_stamp: "Formatted at 2026-06-21 02:39:12 on dist-test-slave-jkn7"
server_key: "9230a342fe147a0dc0a233fe118f9ad6"
server_key_iv: "8966dc58fdedb47bc11decea60074ef2"
server_key_version: "encryptionkey@0"
I20260621 02:39:12.330597 8851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d73f7cdefde44f67984af4849852f28a"
format_stamp: "Formatted at 2026-06-21 02:39:12 on dist-test-slave-jkn7"
server_key: "9230a342fe147a0dc0a233fe118f9ad6"
server_key_iv: "8966dc58fdedb47bc11decea60074ef2"
server_key_version: "encryptionkey@0"
I20260621 02:39:12.335999 8851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:39:12.339334 8866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:12.340590 8851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:39:12.340754 8851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "d73f7cdefde44f67984af4849852f28a"
format_stamp: "Formatted at 2026-06-21 02:39:12 on dist-test-slave-jkn7"
server_key: "9230a342fe147a0dc0a233fe118f9ad6"
server_key_iv: "8966dc58fdedb47bc11decea60074ef2"
server_key_version: "encryptionkey@0"
I20260621 02:39:12.340960 8851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:12.367357 8851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:12.372641 8851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:12.372925 8851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:12.374007 8851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:12.375605 8851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:12.375702 8851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:12.375824 8851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:12.375890 8851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:12.409250 8851 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44227
I20260621 02:39:12.409381 8978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44227 every 8 connection(s)
I20260621 02:39:12.410636 8851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:39:12.419270 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 8851
I20260621 02:39:12.419461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:39:12.419900 388 external_mini_cluster.cc:1428] Setting key b81a8968d43e5027ea8819d43ba5b0fc
Jun 21 02:39:12 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:39:12.434834 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:12.413117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46289 (local address 127.0.97.62:32833)
0621 02:39:12.413378 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:12.413382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:12.414712 (+ 1330us) server_negotiation.cc:407] Connection header received
0621 02:39:12.416487 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:12.416494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:12.416589 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:12.416700 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:12.418198 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:12.418928 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:12.420639 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:12.421313 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:12.421436 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:12.425722 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:12.425727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:12.425730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:12.425759 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:12.431450 (+ 5691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:12.432405 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:12.432409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:12.432411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:12.432536 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:12.433065 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:12.433070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:12.433072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:12.433412 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:12.433527 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:12.434430 (+ 903us) server_negotiation.cc:299] Negotiation successful
0621 02:39:12.434601 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:39:12.436287 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:12.413616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:46289)
0621 02:39:12.414548 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:12.414600 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:12.416228 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:12.416927 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:12.416935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:12.417486 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:12.418035 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:12.418052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:12.419096 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:12.419108 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:12.420474 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:12.420487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:12.421596 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:12.421608 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:12.421930 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:12.422693 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:12.422727 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:12.425507 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:12.431595 (+ 6088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:12.431606 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:12.431626 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:12.432255 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:12.432653 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:12.432660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:12.432667 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:12.432908 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:12.433548 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:12.433558 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:12.434005 (+ 447us) client_negotiation.cc:769] Sending connection context
0621 02:39:12.434311 (+ 306us) client_negotiation.cc:240] Negotiation successful
0621 02:39:12.434850 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":320,"threads_started":1}
I20260621 02:39:12.437790 8979 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32833
I20260621 02:39:12.438108 8979 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:12.438844 8979 heartbeater.cc:511] Master 127.0.97.62:32833 requested a full tablet report, sending...
I20260621 02:39:12.440316 8656 ts_manager.cc:194] Registered new tserver with Master: d73f7cdefde44f67984af4849852f28a (127.0.97.2:44227)
I20260621 02:39:12.441591 8656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46289
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:39:12.467836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32833
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:12.780390 8984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:12.780654 8984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:12.780813 8984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:12.787855 8984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:12.787954 8984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:12.788137 8984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:39:12.797425 8984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:12.798579 8984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:12.799820 8984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:12.812984 8989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:12.812943 8990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:12.813009 8992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:12.814281 8984 server_base.cc:1034] running on GCE node
I20260621 02:39:12.814900 8984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:12.815855 8984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:12.817092 8984 hybrid_clock.cc:648] HybridClock initialized: now 1782009552817031 us; error 54 us; skew 500 ppm
I20260621 02:39:12.817476 8984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:12 dist-test-slave-jkn7 krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:12.820566 8984 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:39:12.822091 8984 webserver.cc:460] Webserver started at http://127.0.97.3:36575/ using document root <none> and password file <none>
I20260621 02:39:12.822716 8984 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:12.822813 8984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:12.823056 8984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:12.825920 8984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "4c68bbaec91d4c32afb93712f72e635e"
format_stamp: "Formatted at 2026-06-21 02:39:12 on dist-test-slave-jkn7"
server_key: "271687ea06c2f792fb7df4fa5788dc94"
server_key_iv: "3b664f8176cbd70e8c89e2deb4d9994d"
server_key_version: "encryptionkey@0"
I20260621 02:39:12.826614 8984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4c68bbaec91d4c32afb93712f72e635e"
format_stamp: "Formatted at 2026-06-21 02:39:12 on dist-test-slave-jkn7"
server_key: "271687ea06c2f792fb7df4fa5788dc94"
server_key_iv: "3b664f8176cbd70e8c89e2deb4d9994d"
server_key_version: "encryptionkey@0"
I20260621 02:39:12.832224 8984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:39:12.835912 8999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:12.837196 8984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:39:12.837383 8984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "4c68bbaec91d4c32afb93712f72e635e"
format_stamp: "Formatted at 2026-06-21 02:39:12 on dist-test-slave-jkn7"
server_key: "271687ea06c2f792fb7df4fa5788dc94"
server_key_iv: "3b664f8176cbd70e8c89e2deb4d9994d"
server_key_version: "encryptionkey@0"
I20260621 02:39:12.837581 8984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:12.858697 8984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:12.862861 8984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:12.863142 8984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:12.864061 8984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:12.865386 8984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:12.865482 8984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:12.865630 8984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:12.865708 8984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:12.897835 8984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32929
I20260621 02:39:12.897921 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32929 every 8 connection(s)
I20260621 02:39:12.899055 8984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:39:12.906945 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 8984
I20260621 02:39:12.907122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:39:12.907640 388 external_mini_cluster.cc:1428] Setting key 0d3cadc02ce8ddb8d157ded07da2f6be
Jun 21 02:39:12 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:39:12.919870 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:12.901642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54987 (local address 127.0.97.62:32833)
0621 02:39:12.901831 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:12.901835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:12.902921 (+ 1086us) server_negotiation.cc:407] Connection header received
0621 02:39:12.904695 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:12.904702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:12.904789 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:12.904897 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:12.906419 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:12.907554 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:12.909241 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:12.909844 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:12.909937 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:12.913715 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:12.913722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:12.913726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:12.913775 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:12.916966 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:12.917778 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:12.917782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:12.917785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:12.917921 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:12.918382 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:12.918386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:12.918388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:12.918668 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:12.918796 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:12.919467 (+ 671us) server_negotiation.cc:299] Negotiation successful
0621 02:39:12.919643 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:39:12.920699 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:12.902078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:54987)
0621 02:39:12.902765 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:12.902807 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:12.904424 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:12.905194 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:12.905202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:12.905737 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:12.906254 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:12.906275 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:12.907763 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:12.907769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:12.909105 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:12.909116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:12.910055 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:12.910074 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:12.910344 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:12.911162 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:12.911183 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:12.913510 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:12.917104 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:12.917110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:12.917121 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:12.917654 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:12.918050 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:12.918053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:12.918056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:12.918242 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:12.918822 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:12.918835 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:12.919116 (+ 281us) client_negotiation.cc:769] Sending connection context
0621 02:39:12.919336 (+ 220us) client_negotiation.cc:240] Negotiation successful
0621 02:39:12.919687 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":283,"threads_started":1}
I20260621 02:39:12.922214 9112 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32833
I20260621 02:39:12.922542 9112 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:12.923276 9112 heartbeater.cc:511] Master 127.0.97.62:32833 requested a full tablet report, sending...
I20260621 02:39:12.924507 8655 ts_manager.cc:194] Registered new tserver with Master: 4c68bbaec91d4c32afb93712f72e635e (127.0.97.3:32929)
I20260621 02:39:12.925455 8655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54987
I20260621 02:39:12.934944 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:39:12.945075 8845 heartbeater.cc:503] Master 127.0.97.62:32833 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260621 02:39:13.056151 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 8717
Jun 21 02:39:13 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:39:13.071550 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:13.057177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.0.97.1:40383)
0621 02:39:13.058119 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:13.058127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:13.058153 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:39:13.058217 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:13.058223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:13.058429 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:13.058569 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:13.059764 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:13.060888 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:13.062076 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:13.062613 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:13.062710 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:13.066367 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:13.066377 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:13.066390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:13.066423 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:13.069002 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:13.069703 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:13.069710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:13.069713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:13.069835 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:13.070221 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:13.070225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:13.070227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:13.070547 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:13.070792 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:13.071089 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 02:39:13.071273 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":638,"threads_started":1}
I20260621 02:39:13.072751 8840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:13.154824 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 8717
I20260621 02:39:13.171196 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 8851
Jun 21 02:39:13 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:39:13.185873 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:13.172042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.0.97.2:44227)
0621 02:39:13.172547 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:13.172555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:13.172581 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:39:13.172644 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:13.172650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:13.172849 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:13.172981 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:13.174191 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:13.175179 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:13.176357 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:13.176891 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:13.176992 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:13.180853 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:13.180864 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:13.180878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:13.180913 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:13.183485 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:13.184064 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:13.184070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:13.184072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:13.184188 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:13.184466 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:13.184468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:13.184470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:13.184791 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:13.184992 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:13.185227 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 02:39:13.185486 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":278,"threads_started":1}
I20260621 02:39:13.187081 8974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:13.267573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 8851
I20260621 02:39:13.283099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 8984
Jun 21 02:39:13 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:39:13.308456 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:13.284011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.0.97.3:32929)
0621 02:39:13.284765 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:13.284772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:13.284799 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:39:13.284859 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:13.284864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:13.285063 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:13.285193 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:13.288875 (+ 3682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:13.290833 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:13.294220 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:13.294765 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:13.294868 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:13.300471 (+ 5603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:13.300486 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:13.300508 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:13.300566 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:13.304999 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:13.305859 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:13.305871 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:13.305876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:13.306054 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:13.306455 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:13.306463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:13.306469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:13.306984 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:13.307336 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:13.307575 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 02:39:13.307906 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":425,"threads_started":1}
I20260621 02:39:13.309751 9107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:13.392531 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 8984
I20260621 02:39:13.407994 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 8624
I20260621 02:39:13.409163 8686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:13.509600 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 8624
I20260621 02:39:13.529902 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32833
--webserver_interface=127.0.97.62
--webserver_port=46053
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:13.858052 9129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:13.858332 9129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:13.858479 9129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:13.865609 9129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:39:13.865737 9129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:13.865820 9129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:13.865891 9129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:39:13.865959 9129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:39:13.875418 9129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=46053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:13.876684 9129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:13.878005 9129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:13.890362 9134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:13.890362 9137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:13.890400 9135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:13.894764 9129 server_base.cc:1034] running on GCE node
I20260621 02:39:13.895581 9129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:13.897054 9129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:13.898306 9129 hybrid_clock.cc:648] HybridClock initialized: now 1782009553898238 us; error 95 us; skew 500 ppm
I20260621 02:39:13.898661 9129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:13 dist-test-slave-jkn7 krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:13.901994 9129 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:39:13.903560 9129 webserver.cc:460] Webserver started at http://127.0.97.62:46053/ using document root <none> and password file <none>
I20260621 02:39:13.904266 9129 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:13.904424 9129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:13.909961 9129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:39:13.912847 9144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:13.914227 9129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:39:13.914422 9129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "aab74b9b7f1b407b93b68456c60bc0c0"
format_stamp: "Formatted at 2026-06-21 02:39:11 on dist-test-slave-jkn7"
server_key: "91166babb7dff7151b2b22604c34c475"
server_key_iv: "16db0b3526798c531c81876441919663"
server_key_version: "encryptionkey@0"
I20260621 02:39:13.915388 9129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:13.944410 9129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:13.948154 9129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:13.948419 9129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:13.966907 9129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32833
I20260621 02:39:13.966931 9195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32833 every 8 connection(s)
I20260621 02:39:13.968320 9129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:39:13.970588 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 9129
I20260621 02:39:13.971354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40383
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:32833
--webserver_port=36345
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:39:13.975090 9196 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:39:13.978272 9196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: Bootstrap starting.
I20260621 02:39:13.996484 9196 log.cc:824] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:14.004782 9196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:39:14.005374 9196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: Bootstrap complete.
I20260621 02:39:14.014132 9196 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } }
I20260621 02:39:14.015000 9196 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab74b9b7f1b407b93b68456c60bc0c0, State: Initialized, Role: FOLLOWER
I20260621 02:39:14.015554 9196 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } }
I20260621 02:39:14.015792 9196 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:39:14.015909 9196 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:39:14.016094 9196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:39:14.020151 9196 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } }
I20260621 02:39:14.020614 9196 leader_election.cc:302] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab74b9b7f1b407b93b68456c60bc0c0; no voters:
I20260621 02:39:14.021231 9196 leader_election.cc:288] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:39:14.021391 9200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:39:14.022585 9200 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [term 2 LEADER]: Becoming Leader. State: Replica: aab74b9b7f1b407b93b68456c60bc0c0, State: Running, Role: LEADER
I20260621 02:39:14.023156 9200 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } }
I20260621 02:39:14.023455 9196 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:39:14.025787 9202 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab74b9b7f1b407b93b68456c60bc0c0. Latest consensus state: current_term: 2 leader_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } } }
I20260621 02:39:14.026223 9202 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:14.026860 9201 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab74b9b7f1b407b93b68456c60bc0c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32833 } } }
I20260621 02:39:14.027194 9201 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:14.042333 9216 catalog_manager.cc:1259] Loaded cluster ID: 9cdbc08b8159428a99a7f4e4285940bf
I20260621 02:39:14.042443 9216 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: loading cluster ID for follower catalog manager: success
I20260621 02:39:14.046234 9216 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: acquiring CA information for follower catalog manager: success
I20260621 02:39:14.048501 9216 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:39:14.048686 9206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:39:14.050447 9206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:39:14.051245 9206 catalog_manager.cc:1259] Loaded cluster ID: 9cdbc08b8159428a99a7f4e4285940bf
I20260621 02:39:14.051355 9206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:39:14.052451 9206 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:39:14.053333 9206 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aab74b9b7f1b407b93b68456c60bc0c0: Loaded TSK: 0
I20260621 02:39:14.054051 9206 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:39:14.329227 9198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:14.329563 9198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:14.329737 9198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:14.337248 9198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:14.337424 9198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:14.337623 9198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:39:14.347803 9198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36345
--tserver_master_addrs=127.0.97.62:32833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:14.349164 9198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:14.350459 9198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:14.365273 9222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:14.365311 9223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:14.365907 9225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:14.366482 9198 server_base.cc:1034] running on GCE node
I20260621 02:39:14.367138 9198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:14.368168 9198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:14.369388 9198 hybrid_clock.cc:648] HybridClock initialized: now 1782009554369339 us; error 41 us; skew 500 ppm
I20260621 02:39:14.369710 9198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:14.372918 9198 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260621 02:39:14.374444 9198 webserver.cc:460] Webserver started at http://127.0.97.1:36345/ using document root <none> and password file <none>
I20260621 02:39:14.375118 9198 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:14.375275 9198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:14.380765 9198 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260621 02:39:14.383711 9232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:14.385210 9198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:39:14.385424 9198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "d69996eb256d464781cd10c5a59d8f11"
format_stamp: "Formatted at 2026-06-21 02:39:11 on dist-test-slave-jkn7"
server_key: "73b94419b5cb14d82fb933e7e5efc230"
server_key_iv: "227b2d554cee78fc6ac58aee71d3c65d"
server_key_version: "encryptionkey@0"
I20260621 02:39:14.386366 9198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:14.414948 9198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:14.419917 9198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:14.420234 9198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:14.421254 9198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:14.422701 9198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:14.422833 9198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:14.422967 9198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:14.423120 9198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:14.458122 9198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40383
I20260621 02:39:14.458217 9344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40383 every 8 connection(s)
I20260621 02:39:14.459524 9198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:39:14.463020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 9198
I20260621 02:39:14.463992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44227
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:32833
--webserver_port=35681
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:14.478015 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:14.462530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:51397)
0621 02:39:14.463341 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:14.463386 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:14.465041 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:14.466971 (+ 1930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:14.466982 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:14.467574 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:14.468089 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.468107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.469776 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.469780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.470950 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.470963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.471977 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.471984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.472255 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.472948 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:14.472970 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:14.476710 (+ 3740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":348,"threads_started":1}
W20260621 02:39:14.478041 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:14.462606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51397 (local address 127.0.97.62:32833)
0621 02:39:14.465756 (+ 3150us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:14.465772 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:14.465822 (+ 50us) server_negotiation.cc:407] Connection header received
0621 02:39:14.466147 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:14.466159 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:14.466472 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:14.466834 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:14.468303 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.469585 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.471120 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.471742 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.471940 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.477071 (+ 5131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51397: BlockingRecv error: recv got EOF from 127.0.97.1:51397 (error 108)
Metrics: {"server-negotiator.queue_time_us":2900,"thread_start_us":2709,"threads_started":1}
W20260621 02:39:14.479094 9345 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32833 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:32833: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:14.488525 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:14.479641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:46759)
0621 02:39:14.479915 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:14.479933 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:14.480036 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:14.480487 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:14.480491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:14.480813 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:14.480948 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.480956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.482560 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.482564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.483437 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.483448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.484524 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.484528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.484680 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.485379 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:14.485406 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:14.488300 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173}
W20260621 02:39:14.488731 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:14.479901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46759 (local address 127.0.97.62:32833)
0621 02:39:14.480108 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:14.480117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:14.480140 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:39:14.480214 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:14.480221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:14.480308 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:14.480434 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:14.481099 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.482398 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.483584 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.484349 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.484501 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.488518 (+ 4017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46759: BlockingRecv error: recv got EOF from 127.0.97.1:46759 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:14.497146 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:14.489204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:50051)
0621 02:39:14.489515 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:14.489529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:14.489612 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:14.489931 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:14.489934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:14.490237 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:14.490386 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.490394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.491785 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.491789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.492707 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.492718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.493637 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.493642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.493781 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.494378 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:14.494394 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:14.496948 (+ 2554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226}
W20260621 02:39:14.497504 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:14.489365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50051 (local address 127.0.97.62:32833)
0621 02:39:14.489520 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:14.489526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:14.489571 (+ 45us) server_negotiation.cc:407] Connection header received
0621 02:39:14.489703 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:14.489708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:14.489806 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:14.489898 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:14.490521 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.491617 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.492845 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.493481 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.493600 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.497207 (+ 3607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50051: BlockingRecv error: recv got EOF from 127.0.97.1:50051 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260621 02:39:14.497510 9345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:39:14.788017 9349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:14.788311 9349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:14.788404 9349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:14.795745 9349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:14.795863 9349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:14.795979 9349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:39:14.805637 9349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35681
--tserver_master_addrs=127.0.97.62:32833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:14.806948 9349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:14.808182 9349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:14.821798 9354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:14.821751 9357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:14.821714 9355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:14.827032 9349 server_base.cc:1034] running on GCE node
I20260621 02:39:14.827739 9349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:14.828770 9349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:14.830005 9349 hybrid_clock.cc:648] HybridClock initialized: now 1782009554829964 us; error 44 us; skew 500 ppm
I20260621 02:39:14.830338 9349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:14.833617 9349 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260621 02:39:14.835253 9349 webserver.cc:460] Webserver started at http://127.0.97.2:35681/ using document root <none> and password file <none>
I20260621 02:39:14.835973 9349 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:14.836125 9349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:14.842711 9349 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260621 02:39:14.845500 9364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:14.846870 9349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:39:14.847069 9349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "d73f7cdefde44f67984af4849852f28a"
format_stamp: "Formatted at 2026-06-21 02:39:12 on dist-test-slave-jkn7"
server_key: "9230a342fe147a0dc0a233fe118f9ad6"
server_key_iv: "8966dc58fdedb47bc11decea60074ef2"
server_key_version: "encryptionkey@0"
I20260621 02:39:14.848024 9349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:14.864965 9349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:14.868954 9349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:14.869240 9349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:14.870246 9349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:14.871552 9349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:14.871686 9349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:14.871815 9349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:14.871905 9349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:14.904901 9349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44227
I20260621 02:39:14.904937 9476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44227 every 8 connection(s)
I20260621 02:39:14.906345 9349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:39:14.906757 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 9349
I20260621 02:39:14.907513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:32929
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:32833
--webserver_port=36575
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:39:14.924147 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:14.909042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59543 (local address 127.0.97.62:32833)
0621 02:39:14.909255 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:14.909262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:14.910550 (+ 1288us) server_negotiation.cc:407] Connection header received
0621 02:39:14.913372 (+ 2822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:14.913377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:14.913442 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:14.913527 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:14.915156 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.916381 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.917959 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.918740 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.918871 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.923806 (+ 4935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59543: BlockingRecv error: recv got EOF from 127.0.97.2:59543 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:39:14.924610 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:14.909533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:59543)
0621 02:39:14.910359 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:14.910411 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:14.913038 (+ 2627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:14.913788 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:14.913802 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:14.914439 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:14.914971 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.914987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.916530 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.916534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.917821 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.917832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.918979 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.918985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.919233 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.919942 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:14.919963 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:14.923510 (+ 3547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":306,"threads_started":1}
W20260621 02:39:14.925344 9477 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32833 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:32833: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:14.933672 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:14.926004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:35603)
0621 02:39:14.926178 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:14.926197 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:14.926298 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:14.926624 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:14.926629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:14.926950 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:14.927073 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.927080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.928115 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.928121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.929005 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.929015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.929837 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.929842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.930005 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.930593 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:14.930609 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:14.933396 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260621 02:39:14.933782 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:14.926080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35603 (local address 127.0.97.62:32833)
0621 02:39:14.926233 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:14.926240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:14.926268 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:39:14.926432 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:14.926437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:14.926505 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:14.926604 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:14.927208 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.927942 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.929153 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.929673 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.929829 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.933585 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35603: BlockingRecv error: recv got EOF from 127.0.97.2:35603 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:14 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:14.942086 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:14.934373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:49119)
0621 02:39:14.934564 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:14.934579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:14.934654 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:14.934970 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:14.934973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:14.935235 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:14.935370 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.935377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.936692 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.936696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.937634 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:14.937644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.938524 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.938528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:14.938646 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.939243 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:14.939265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:14.941880 (+ 2615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 02:39:14.942417 9477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:39:14.942684 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:14.934468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49119 (local address 127.0.97.62:32833)
0621 02:39:14.934619 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:14.934625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:14.934643 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:39:14.934732 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:14.934737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:14.934802 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:14.934894 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:14.935485 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.936560 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.937751 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:14.938379 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:14.938512 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:14.942101 (+ 3589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49119: BlockingRecv error: recv got EOF from 127.0.97.2:49119 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260621 02:39:15.234709 9480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:15.235025 9480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:15.235126 9480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:15.242532 9480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:15.242656 9480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:15.242769 9480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:39:15.252882 9480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:32929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36575
--tserver_master_addrs=127.0.97.62:32833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:15.254262 9480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:15.255504 9480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:15.268940 9486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:15.268988 9488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:15.268884 9485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:15.269901 9480 server_base.cc:1034] running on GCE node
I20260621 02:39:15.270569 9480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:15.271560 9480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:15.272763 9480 hybrid_clock.cc:648] HybridClock initialized: now 1782009555272718 us; error 46 us; skew 500 ppm
I20260621 02:39:15.273088 9480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:15.276173 9480 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260621 02:39:15.277747 9480 webserver.cc:460] Webserver started at http://127.0.97.3:36575/ using document root <none> and password file <none>
I20260621 02:39:15.278471 9480 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:15.278632 9480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:15.284324 9480 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:39:15.287302 9495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:15.288655 9480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:39:15.288852 9480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "4c68bbaec91d4c32afb93712f72e635e"
format_stamp: "Formatted at 2026-06-21 02:39:12 on dist-test-slave-jkn7"
server_key: "271687ea06c2f792fb7df4fa5788dc94"
server_key_iv: "3b664f8176cbd70e8c89e2deb4d9994d"
server_key_version: "encryptionkey@0"
I20260621 02:39:15.289870 9480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:15.304816 9480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:15.308633 9480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:15.308917 9480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:15.309906 9480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:15.311167 9480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:15.311290 9480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:15.311417 9480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:15.311507 9480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:15.345557 9480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32929
I20260621 02:39:15.345608 9607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32929 every 8 connection(s)
I20260621 02:39:15.347002 9480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:39:15.351310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 9480
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:39:15.367725 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:15.349509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48113 (local address 127.0.97.62:32833)
0621 02:39:15.349689 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:15.349694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:15.350850 (+ 1156us) server_negotiation.cc:407] Connection header received
0621 02:39:15.353542 (+ 2692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:15.353551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:15.353680 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:15.353797 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:15.356211 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.357635 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.359914 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.360725 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.360863 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.367485 (+ 6622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48113: BlockingRecv error: recv got EOF from 127.0.97.3:48113 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":62}
I20260621 02:39:15.368677 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:15.349948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:48113)
0621 02:39:15.350677 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:15.350714 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:15.353177 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:15.354269 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:15.354284 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:15.355104 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:15.355977 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.356006 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.357804 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.357812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.359699 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.359713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.361476 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.361483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.361860 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.362616 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:15.362645 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:15.367322 (+ 4677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":302,"threads_started":1}
W20260621 02:39:15.369426 9608 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32833 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:32833: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260621 02:39:15.372177 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:15.352459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.0.97.62:32833)
0621 02:39:15.353427 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:15.353435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:15.353455 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:39:15.353510 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:15.353515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:15.353590 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:15.356015 (+ 2425us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:15.356224 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.357495 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.360172 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.360944 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.361055 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.364425 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:15.364439 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:15.364462 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:15.364506 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:15.368489 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:15.369375 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:15.369385 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:15.369396 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:15.369638 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:15.370078 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:15.370083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:15.370085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:15.370687 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:15.370988 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:15.371495 (+ 507us) server_negotiation.cc:299] Negotiation successful
0621 02:39:15.371873 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":854,"thread_start_us":717,"threads_started":1}
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:15.377825 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:15.370060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:48015)
0621 02:39:15.370229 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:15.370251 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:15.370326 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:15.370632 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:15.370636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:15.370931 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:15.371057 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.371065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.372237 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.372242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.373287 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.373301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.374043 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.374049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.374208 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.374747 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:15.374772 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:15.377636 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:39:15.378047 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:15.370180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48015 (local address 127.0.97.62:32833)
0621 02:39:15.370334 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:15.370339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:15.370356 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:39:15.370400 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:15.370404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:15.370463 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:15.370560 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:15.371214 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.372097 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.373460 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.373892 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.373991 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.377811 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48015: BlockingRecv error: recv got EOF from 127.0.97.3:48015 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:15.385215 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:15.378486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:56465)
0621 02:39:15.378637 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:15.378654 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:15.378730 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:15.378972 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:15.378975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:15.379208 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:15.379336 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.379342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.380189 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.380192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.381050 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.381060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.381813 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.381817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.381936 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.382393 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:15.382409 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:15.385026 (+ 2617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 02:39:15.385524 9608 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:39:15.385571 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:15.378576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56465 (local address 127.0.97.62:32833)
0621 02:39:15.378764 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:15.378768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:15.378781 (+ 13us) server_negotiation.cc:407] Connection header received
0621 02:39:15.378817 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:15.378820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:15.378866 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:15.378939 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:15.379441 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.380087 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.381168 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.381662 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.381764 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.385221 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56465: BlockingRecv error: recv got EOF from 127.0.97.3:56465 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:15.506493 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:15.498068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:51427)
0621 02:39:15.498561 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:15.498576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:15.498686 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:15.498953 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:15.498955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:15.499395 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:15.499576 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.499583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.500582 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.500587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.501581 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.501604 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.502311 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.502315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.502430 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.503084 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:15.503101 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:15.506271 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":266,"threads_started":1}
W20260621 02:39:15.506664 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:15.498244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51427 (local address 127.0.97.62:32833)
0621 02:39:15.498429 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:15.498434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:15.498646 (+ 212us) server_negotiation.cc:407] Connection header received
0621 02:39:15.498775 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:15.498779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:15.498835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:15.498913 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:15.499717 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.500431 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.501735 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.502157 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.502258 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.506461 (+ 4203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51427: BlockingRecv error: recv got EOF from 127.0.97.1:51427 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:15 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:15.951555 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:15.943021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:55607)
0621 02:39:15.943464 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:15.943484 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:15.943596 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:15.943877 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:15.943880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:15.944316 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:15.944504 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.944512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.945514 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.945520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.946463 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:15.946474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.947192 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.947198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:15.947365 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.948020 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:15.948048 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:15.951209 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":240,"threads_started":1}
W20260621 02:39:15.951637 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:15.943144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55607 (local address 127.0.97.62:32833)
0621 02:39:15.943393 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:15.943399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:15.943560 (+ 161us) server_negotiation.cc:407] Connection header received
0621 02:39:15.943688 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:15.943692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:15.943749 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:15.943829 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:15.944618 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.945348 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.946621 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:15.947016 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:15.947149 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:15.951394 (+ 4245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55607: BlockingRecv error: recv got EOF from 127.0.97.2:55607 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Jun 21 02:39:16 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:16 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:16.394711 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:16.386212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:54505)
0621 02:39:16.386754 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:16.386770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:16.386928 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:16.387372 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:16.387377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:16.387958 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:16.388201 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:16.388234 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.389377 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.389386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:16.390349 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:16.390359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.391048 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.391054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:16.391179 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:16.391781 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:16.391798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:16.394487 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":293,"threads_started":1}
W20260621 02:39:16.394884 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:16.386234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54505 (local address 127.0.97.62:32833)
0621 02:39:16.386388 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:16.386393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:16.386865 (+ 472us) server_negotiation.cc:407] Connection header received
0621 02:39:16.387064 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:16.387068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:16.387193 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:16.387321 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:16.388363 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.389190 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.390477 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.390866 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.390983 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:16.394661 (+ 3678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54505: BlockingRecv error: recv got EOF from 127.0.97.3:54505 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 02:39:16 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:16 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:16.514912 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:16.507352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:59945)
0621 02:39:16.507810 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:16.507826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:16.507932 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:16.508227 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:16.508230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:16.508582 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:16.508731 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:16.508738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.509731 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.509735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:16.510608 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:16.510619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.511318 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.511322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:16.511429 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:16.512059 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:16.512077 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:16.514674 (+ 2597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":256,"threads_started":1}
W20260621 02:39:16.515133 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:16.507383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59945 (local address 127.0.97.62:32833)
0621 02:39:16.507597 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:16.507607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:16.507926 (+ 319us) server_negotiation.cc:407] Connection header received
0621 02:39:16.508029 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:16.508032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:16.508093 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:16.508179 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:16.508846 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.509592 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.510758 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.511180 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.511286 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:16.514899 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59945: BlockingRecv error: recv got EOF from 127.0.97.1:59945 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 02:39:16 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:16 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:16.960739 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:16.952561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:42705)
0621 02:39:16.953095 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:16.953121 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:16.953263 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:16.953645 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:16.953650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:16.954077 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:16.954220 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:16.954228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.955229 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.955238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:16.956134 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:16.956145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.956854 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.956860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:16.957019 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:16.957687 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:16.957704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:16.960506 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":295,"threads_started":1}
W20260621 02:39:16.960911 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:16.952630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42705 (local address 127.0.97.62:32833)
0621 02:39:16.952836 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:16.952843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:16.953227 (+ 384us) server_negotiation.cc:407] Connection header received
0621 02:39:16.953435 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:16.953444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:16.953504 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:16.953614 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:16.954369 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.955091 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.956285 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:16.956705 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:16.956840 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:16.960708 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42705: BlockingRecv error: recv got EOF from 127.0.97.2:42705 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 02:39:17 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:17 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:17.403334 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:17.395598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:57709)
0621 02:39:17.396062 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:17.396078 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:17.396187 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:17.396479 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:17.396482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:17.396856 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:17.396997 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:17.397004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.398047 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.398051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:17.398964 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:17.398977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.399621 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.399626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:17.399734 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:17.400284 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:17.400301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:17.403103 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
W20260621 02:39:17.403539 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:17.395766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57709 (local address 127.0.97.62:32833)
0621 02:39:17.395983 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:17.395989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:17.396160 (+ 171us) server_negotiation.cc:407] Connection header received
0621 02:39:17.396282 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:17.396286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:17.396358 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:17.396435 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:17.397134 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.397888 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.399091 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.399482 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.399571 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:17.403284 (+ 3713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57709: BlockingRecv error: recv got EOF from 127.0.97.3:57709 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 21 02:39:17 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:17 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:17.528482 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:17.515885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:54097)
0621 02:39:17.516376 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:17.516391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:17.516551 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:17.516892 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:17.516896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:17.517260 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:17.517444 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:17.517453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.518494 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.518501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:17.519585 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:17.519597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.520330 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.520336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:17.520509 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:17.525268 (+ 4759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:17.525321 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:17.528246 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":263,"threads_started":1}
W20260621 02:39:17.528764 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:17.515960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54097 (local address 127.0.97.62:32833)
0621 02:39:17.516203 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:17.516211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:17.516482 (+ 271us) server_negotiation.cc:407] Connection header received
0621 02:39:17.516687 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:17.516693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:17.516758 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:17.516867 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:17.517596 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.518333 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.519763 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.520182 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.520319 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:17.528463 (+ 8144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54097: BlockingRecv error: recv got EOF from 127.0.97.1:54097 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 02:39:17 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:17 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:17.969724 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:17.961758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:37111)
0621 02:39:17.962245 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:17.962266 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:17.962381 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:17.962655 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:17.962658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:17.962980 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:17.963135 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:17.963143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.964112 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.964115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:17.964997 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:17.965007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.965641 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.965645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:17.965762 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:17.966348 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:17.966368 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:17.969487 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":269,"threads_started":1}
W20260621 02:39:17.969987 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:17.961787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37111 (local address 127.0.97.62:32833)
0621 02:39:17.961956 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:17.961960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:17.962339 (+ 379us) server_negotiation.cc:407] Connection header received
0621 02:39:17.962479 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:17.962482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:17.962539 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:17.962619 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:17.963265 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.963998 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.965114 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:17.965527 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:17.965616 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:17.969754 (+ 4138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37111: BlockingRecv error: recv got EOF from 127.0.97.2:37111 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 02:39:18 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:18 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:18.412031 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:18.404295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:36817)
0621 02:39:18.404800 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:18.404816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:18.404934 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:18.405218 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:18.405222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:18.405562 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:18.405736 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:18.405744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.406763 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.406767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:18.407636 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:18.407646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.408283 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.408286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:18.408402 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:18.409040 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:18.409056 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:18.411801 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":280,"threads_started":1}
W20260621 02:39:18.412235 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:18.404400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36817 (local address 127.0.97.62:32833)
0621 02:39:18.404569 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:18.404573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:18.404886 (+ 313us) server_negotiation.cc:407] Connection header received
0621 02:39:18.405017 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:18.405020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:18.405098 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:18.405179 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:18.405912 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.406627 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.407766 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.408166 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.408260 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:18.411988 (+ 3728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36817: BlockingRecv error: recv got EOF from 127.0.97.3:36817 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 21 02:39:18 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:18 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:18.538080 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:18.529499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:53735)
0621 02:39:18.530065 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:18.530082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:18.530210 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:18.530508 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:18.530511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:18.530866 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:18.531012 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:18.531021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.532013 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.532016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:18.532956 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:18.532966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.533688 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.533692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:18.533862 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:18.534538 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:18.534555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:18.537751 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":317,"threads_started":1}
W20260621 02:39:18.538239 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:18.529539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53735 (local address 127.0.97.62:32833)
0621 02:39:18.529803 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:18.529813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:18.530173 (+ 360us) server_negotiation.cc:407] Connection header received
0621 02:39:18.530319 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:18.530323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:18.530382 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:18.530464 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:18.531178 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.531903 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.533107 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.533558 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.533660 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:18.537977 (+ 4317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53735: BlockingRecv error: recv got EOF from 127.0.97.1:53735 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 21 02:39:18 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:18 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:18.978962 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:18.970728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:34327)
0621 02:39:18.971366 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:18.971385 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:18.971534 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:18.971866 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:18.971870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:18.972239 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:18.972376 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:18.972385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.973398 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.973404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:18.974343 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:18.974355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.975031 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.975036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:18.975224 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:18.975830 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:18.975847 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:18.978643 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":398,"threads_started":1}
W20260621 02:39:18.979043 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:18.970795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34327 (local address 127.0.97.62:32833)
0621 02:39:18.971021 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:18.971028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:18.971467 (+ 439us) server_negotiation.cc:407] Connection header received
0621 02:39:18.971666 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:18.971675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:18.971735 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:18.971835 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:18.972513 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.973246 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.974485 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:18.974888 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:18.975019 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:18.978832 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34327: BlockingRecv error: recv got EOF from 127.0.97.2:34327 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 02:39:19 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:19 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:19.421455 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:19.413038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:54803)
0621 02:39:19.413598 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:19.413615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:19.413733 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:19.414086 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:19.414090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:19.414474 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:19.414623 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:19.414631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.415608 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.415611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:19.416549 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:19.416559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.417214 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.417219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:19.417453 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:19.418070 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:19.418087 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:19.421089 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":308,"threads_started":1}
W20260621 02:39:19.421541 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:19.413166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54803 (local address 127.0.97.62:32833)
0621 02:39:19.413394 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:19.413399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:19.413694 (+ 295us) server_negotiation.cc:407] Connection header received
0621 02:39:19.413836 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:19.413840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:19.413912 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:19.414001 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:19.414752 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.415495 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.416670 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.417050 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.417139 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:19.421278 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54803: BlockingRecv error: recv got EOF from 127.0.97.3:54803 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 21 02:39:19 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:19 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:19.547060 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:19.539113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:42109)
0621 02:39:19.539631 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:19.539646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:19.539754 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:19.540026 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:19.540029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:19.540392 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:19.540547 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:19.540555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.541621 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.541625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:19.542617 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:19.542628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.543341 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.543345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:19.543464 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:19.544021 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:19.544038 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:19.546723 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":259,"threads_started":1}
W20260621 02:39:19.547106 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:19.539174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42109 (local address 127.0.97.62:32833)
0621 02:39:19.539372 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:19.539376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:19.539722 (+ 346us) server_negotiation.cc:407] Connection header received
0621 02:39:19.539846 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:19.539850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:19.539905 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:19.539987 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:19.540665 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.541507 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.542775 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.543192 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.543290 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:19.546896 (+ 3606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42109: BlockingRecv error: recv got EOF from 127.0.97.1:42109 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 21 02:39:19 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:19 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:19.988479 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:19.980063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:53117)
0621 02:39:19.980612 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:19.980633 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:19.980743 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:19.981042 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:19.981045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:19.981488 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:19.981660 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:19.981670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.982662 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.982666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:19.983571 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:19.983581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.984289 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.984293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:19.984405 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:19.984975 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:19.984992 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:19.988139 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":277,"threads_started":1}
W20260621 02:39:19.988726 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:19.980228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53117 (local address 127.0.97.62:32833)
0621 02:39:19.980435 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:19.980439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:19.980709 (+ 270us) server_negotiation.cc:407] Connection header received
0621 02:39:19.980854 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:19.980858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:19.980916 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:19.980998 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:19.981832 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.982547 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.983717 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:19.984160 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:19.984258 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:19.988341 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53117: BlockingRecv error: recv got EOF from 127.0.97.2:53117 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 21 02:39:20 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:20 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:20.431084 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:20.422511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:47977)
0621 02:39:20.423038 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:20.423055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:20.423161 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:20.423461 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:20.423464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:20.423972 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:20.424133 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:20.424142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.425142 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.425146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:20.426079 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:20.426089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.426846 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.426851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:20.426964 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:20.427525 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:20.427548 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:20.430741 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":271,"threads_started":1}
W20260621 02:39:20.431130 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:20.422620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47977 (local address 127.0.97.62:32833)
0621 02:39:20.422797 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:20.422801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:20.423155 (+ 354us) server_negotiation.cc:407] Connection header received
0621 02:39:20.423260 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:20.423263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:20.423339 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:20.423425 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:20.424262 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.425010 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.426236 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.426697 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.426801 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:20.430898 (+ 4097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47977: BlockingRecv error: recv got EOF from 127.0.97.3:47977 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 02:39:20 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:20 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:20.555981 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:20.547920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:32791)
0621 02:39:20.548464 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:20.548480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:20.548589 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:20.548885 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:20.548888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:20.549243 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:20.549421 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:20.549428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.550430 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.550434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:20.551399 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:20.551411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.552224 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.552229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:20.552418 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:20.553045 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:20.553062 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:20.555760 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":304,"threads_started":1}
W20260621 02:39:20.556187 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:20.548002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32791 (local address 127.0.97.62:32833)
0621 02:39:20.548195 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:20.548200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:20.548580 (+ 380us) server_negotiation.cc:407] Connection header received
0621 02:39:20.548697 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:20.548701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:20.548760 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:20.548842 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:20.549552 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.550317 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.551561 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.552015 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.552123 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:20.555958 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:32791: BlockingRecv error: recv got EOF from 127.0.97.1:32791 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 21 02:39:20 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:20 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:20.997814 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:20.989520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:57825)
0621 02:39:20.990103 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:20.990135 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:20.990258 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:20.990544 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:20.990548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:20.990909 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:20.991054 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:20.991062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.991993 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.991997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:20.992968 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:20.992979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.993739 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.993745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:20.993925 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:20.994535 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:20.994554 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:20.997481 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":302,"threads_started":1}
W20260621 02:39:20.997944 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:20.989558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57825 (local address 127.0.97.62:32833)
0621 02:39:20.989762 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:20.989772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:20.990205 (+ 433us) server_negotiation.cc:407] Connection header received
0621 02:39:20.990360 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:20.990363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:20.990424 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:20.990503 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:20.991188 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.991874 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.993087 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:20.993584 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:20.993675 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:20.997696 (+ 4021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57825: BlockingRecv error: recv got EOF from 127.0.97.2:57825 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 02:39:21 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:21 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:21.440529 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:21.432105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:45995)
0621 02:39:21.432688 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:21.432704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:21.432835 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:21.433198 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:21.433202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:21.433610 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:21.433773 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:21.433782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:21.434929 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:21.434934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:21.435919 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:21.435931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:21.436643 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:21.436652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:21.436796 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:21.437504 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:21.437531 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:21.440315 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":325,"threads_started":1}
W20260621 02:39:21.440821 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:21.432206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45995 (local address 127.0.97.62:32833)
0621 02:39:21.432410 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:21.432414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:21.432829 (+ 415us) server_negotiation.cc:407] Connection header received
0621 02:39:21.432940 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:21.432945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:21.433017 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:21.433113 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:21.433921 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:21.434784 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:21.436049 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:21.436478 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:21.436571 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:21.440526 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45995: BlockingRecv error: recv got EOF from 127.0.97.3:45995 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 21 02:39:21 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:21 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:21.565577 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:21.556995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:55873)
0621 02:39:21.557692 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:21.557709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:21.557901 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:21.558255 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:21.558260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:21.558791 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:21.558958 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:21.558967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:21.559979 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:21.559986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:21.560896 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:21.560907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:21.561672 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:21.561679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:21.561866 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:21.562470 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:21.562486 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:21.565210 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":315,"threads_started":1}
W20260621 02:39:21.565667 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:21.557175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55873 (local address 127.0.97.62:32833)
0621 02:39:21.557456 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:21.557465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:21.557807 (+ 342us) server_negotiation.cc:407] Connection header received
0621 02:39:21.558039 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:21.558045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:21.558113 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:21.558222 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:21.559103 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:21.559837 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:21.561039 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:21.561501 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:21.561628 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:21.565444 (+ 3816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55873: BlockingRecv error: recv got EOF from 127.0.97.1:55873 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 02:39:22 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:22 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:22.007925 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:21.998959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:37595)
0621 02:39:21.999635 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:21.999656 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:21.999793 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:22.000137 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:22.000140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:22.000515 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:22.000681 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:22.000689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.001811 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.001826 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:22.002741 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:22.002752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.003610 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.003617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:22.003807 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:22.004491 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:22.004516 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:22.007567 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":327,"threads_started":1}
W20260621 02:39:22.008025 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:21.999079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37595 (local address 127.0.97.62:32833)
0621 02:39:21.999277 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:21.999282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:21.999751 (+ 469us) server_negotiation.cc:407] Connection header received
0621 02:39:21.999916 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:21.999920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:22.000005 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:22.000093 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:22.000859 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.001633 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.002896 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.003404 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.003549 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:22.007773 (+ 4224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37595: BlockingRecv error: recv got EOF from 127.0.97.2:37595 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 21 02:39:22 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:22 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:22.450012 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:22.441533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:50805)
0621 02:39:22.442182 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:22.442205 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:22.442337 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:22.442666 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:22.442671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:22.443183 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:22.443411 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:22.443420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.444380 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.444383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:22.445272 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:22.445299 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.445971 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.445975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:22.446077 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:22.446691 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:22.446708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:22.449667 (+ 2959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":246,"threads_started":1}
W20260621 02:39:22.450050 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:22.441570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50805 (local address 127.0.97.62:32833)
0621 02:39:22.441772 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:22.441776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:22.442301 (+ 525us) server_negotiation.cc:407] Connection header received
0621 02:39:22.442461 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:22.442464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:22.442524 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:22.442601 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:22.443538 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.444254 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.445427 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.445846 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.445939 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:22.449837 (+ 3898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50805: BlockingRecv error: recv got EOF from 127.0.97.3:50805 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 21 02:39:22 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:22 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:22.577677 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:22.566624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:50423)
0621 02:39:22.567162 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:22.567179 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:22.567292 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:22.567583 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:22.567586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:22.567974 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:22.568129 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:22.568137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.569143 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.569147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:22.570262 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:22.570275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.570982 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.570987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:22.571104 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:22.574783 (+ 3679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:22.574800 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:22.577457 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":310,"threads_started":1}
W20260621 02:39:22.577898 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:22.566843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50423 (local address 127.0.97.62:32833)
0621 02:39:22.567060 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:22.567064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:22.567257 (+ 193us) server_negotiation.cc:407] Connection header received
0621 02:39:22.567397 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:22.567401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:22.567460 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:22.567544 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:22.568261 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.568988 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.570398 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:22.570797 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:22.570893 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:22.577657 (+ 6764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50423: BlockingRecv error: recv got EOF from 127.0.97.1:50423 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 21 02:39:23 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:23 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:23.016979 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:23.008876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:33787)
0621 02:39:23.009385 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:23.009407 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:23.009518 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:23.009804 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:23.009807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:23.010196 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:23.010343 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:23.010350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.011445 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.011449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:23.012344 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:23.012354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.013024 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.013028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:23.013139 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:23.013738 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:23.013756 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:23.016654 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":268,"threads_started":1}
W20260621 02:39:23.017024 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:23.008932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33787 (local address 127.0.97.62:32833)
0621 02:39:23.009107 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:23.009111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:23.009479 (+ 368us) server_negotiation.cc:407] Connection header received
0621 02:39:23.009612 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:23.009615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:23.009673 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:23.009758 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:23.010460 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.011308 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.012462 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.012877 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.012964 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:23.016817 (+ 3853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33787: BlockingRecv error: recv got EOF from 127.0.97.2:33787 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 21 02:39:23 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:23 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:23.458971 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:23.451054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:48843)
0621 02:39:23.451583 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:23.451599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:23.451754 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:23.452171 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:23.452175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:23.452540 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:23.452697 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:23.452705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.453723 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.453727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:23.454616 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:23.454626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.455257 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.455261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:23.455375 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:23.455939 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:23.455955 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:23.458759 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":296,"threads_started":1}
W20260621 02:39:23.459226 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:23.451151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48843 (local address 127.0.97.62:32833)
0621 02:39:23.451348 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:23.451355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:23.451676 (+ 321us) server_negotiation.cc:407] Connection header received
0621 02:39:23.451939 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:23.451944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:23.452025 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:23.452116 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:23.452815 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.453581 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.454746 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.455115 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.455201 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:23.458979 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48843: BlockingRecv error: recv got EOF from 127.0.97.3:48843 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 02:39:23 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:23 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:23.587021 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:23.578631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:44915)
0621 02:39:23.579154 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:23.579174 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:23.579316 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:23.579648 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:23.579653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:23.580071 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:23.580231 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:23.580239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.581214 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.581219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:23.582170 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:23.582181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.582888 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.582894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:23.583088 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:23.583650 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:23.583667 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:23.586777 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":370,"threads_started":1}
W20260621 02:39:23.587193 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:23.578750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44915 (local address 127.0.97.62:32833)
0621 02:39:23.578993 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:23.578999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:23.579246 (+ 247us) server_negotiation.cc:407] Connection header received
0621 02:39:23.579452 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:23.579457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:23.579522 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:23.579651 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:23.580376 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.581081 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.582324 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:23.582739 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:23.582873 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:23.586978 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44915: BlockingRecv error: recv got EOF from 127.0.97.1:44915 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 21 02:39:24 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:24 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:24.026423 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:24.017976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:34855)
0621 02:39:24.018489 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:24.018509 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:24.018665 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:24.018992 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:24.018997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:24.019665 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:24.019891 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:24.019901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.020919 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.020927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:24.021895 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:24.021906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.022606 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.022612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:24.022777 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:24.023332 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:24.023349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:24.026178 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":274,"threads_started":1}
W20260621 02:39:24.026633 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:24.018047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34855 (local address 127.0.97.62:32833)
0621 02:39:24.018292 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:24.018300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:24.018594 (+ 294us) server_negotiation.cc:407] Connection header received
0621 02:39:24.018787 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:24.018792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:24.018864 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:24.018964 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:24.020029 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.020789 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.022040 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.022459 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.022592 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:24.026407 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34855: BlockingRecv error: recv got EOF from 127.0.97.2:34855 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 21 02:39:24 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:24 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:24.468806 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:24.460031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:56471)
0621 02:39:24.460534 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:24.460550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:24.460707 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:24.461050 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:24.461055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:24.461469 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:24.461851 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:24.461861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.462855 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.462862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:24.464053 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:24.464067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.464999 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.465007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:24.465144 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:24.465708 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:24.465734 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:24.468560 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":282,"threads_started":1}
W20260621 02:39:24.469014 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:24.460107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56471 (local address 127.0.97.62:32833)
0621 02:39:24.460313 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:24.460321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:24.460637 (+ 316us) server_negotiation.cc:407] Connection header received
0621 02:39:24.460847 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:24.460854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:24.460918 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:24.461020 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:24.462011 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.462720 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.464177 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.464834 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.464963 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:24.468802 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56471: BlockingRecv error: recv got EOF from 127.0.97.3:56471 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 21 02:39:24 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:24 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:24.596606 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:24.588015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:59007)
0621 02:39:24.588511 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:24.588526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:24.588676 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:24.589018 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:24.589023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:24.589436 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:24.589624 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:24.589640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.590666 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.590672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:24.591610 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:24.591620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.592338 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.592344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:24.592521 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:24.593163 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:24.593181 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:24.596354 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":282,"threads_started":1}
W20260621 02:39:24.596756 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:24.588069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59007 (local address 127.0.97.62:32833)
0621 02:39:24.588299 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:24.588308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:24.588607 (+ 299us) server_negotiation.cc:407] Connection header received
0621 02:39:24.588793 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:24.588799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:24.588871 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:24.588990 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:24.589789 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.590537 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.591741 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:24.592173 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:24.592320 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:24.596537 (+ 4217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59007: BlockingRecv error: recv got EOF from 127.0.97.1:59007 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 02:39:25 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:25 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:25.036494 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:25.027416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:32815)
0621 02:39:25.027974 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:25.028000 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:25.028156 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:25.028520 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:25.028525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:25.028928 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:25.029075 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:25.029084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.030222 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.030227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:25.031559 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:25.031571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.032340 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.032345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:25.032463 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:25.033184 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:25.033202 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:25.036280 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":340,"threads_started":1}
W20260621 02:39:25.036821 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:25.027532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32815 (local address 127.0.97.62:32833)
0621 02:39:25.027781 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:25.027789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:25.028092 (+ 303us) server_negotiation.cc:407] Connection header received
0621 02:39:25.028308 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:25.028313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:25.028380 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:25.028486 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:25.029242 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.030055 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.031701 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.032180 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.032298 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:25.036493 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:32815: BlockingRecv error: recv got EOF from 127.0.97.2:32815 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 21 02:39:25 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:25 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:25.477993 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:25.469842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:60413)
0621 02:39:25.470363 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:25.470379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:25.470523 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:25.470970 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:25.470976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:25.471463 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:25.471606 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:25.471615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.472605 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.472609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:25.473544 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:25.473554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.474310 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.474315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:25.474437 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:25.475106 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:25.475124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:25.477781 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":269,"threads_started":1}
W20260621 02:39:25.478181 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:25.470042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60413 (local address 127.0.97.62:32833)
0621 02:39:25.470288 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:25.470296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:25.470467 (+ 171us) server_negotiation.cc:407] Connection header received
0621 02:39:25.470653 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:25.470658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:25.470755 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:25.470854 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:25.471749 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.472476 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.473702 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.474132 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.474268 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:25.477965 (+ 3697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60413: BlockingRecv error: recv got EOF from 127.0.97.3:60413 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 21 02:39:25 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:25 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:25.605643 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:25.597441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:37731)
0621 02:39:25.597900 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:25.597916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:25.598024 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:25.598349 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:25.598352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:25.598725 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:25.598896 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:25.598904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.599877 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.599880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:25.600751 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:25.600761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.601659 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.601665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:25.601885 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:25.602634 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:25.602654 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:25.605413 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":263,"threads_started":1}
W20260621 02:39:25.605808 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:25.597524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37731 (local address 127.0.97.62:32833)
0621 02:39:25.597812 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:25.597819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:25.597996 (+ 177us) server_negotiation.cc:407] Connection header received
0621 02:39:25.598161 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:25.598165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:25.598226 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:25.598313 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:25.599062 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.599767 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.600895 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:25.601490 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:25.601636 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:25.605584 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37731: BlockingRecv error: recv got EOF from 127.0.97.1:37731 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 21 02:39:26 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:26 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:26.045833 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:26.037428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:40649)
0621 02:39:26.038131 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:26.038151 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:26.038312 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:26.038704 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:26.038709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:26.039116 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:26.039302 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:26.039311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.040332 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.040339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:26.041201 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:26.041211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.042014 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.042019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:26.042191 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:26.042817 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:26.042834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:26.045497 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":462,"threads_started":1}
W20260621 02:39:26.045926 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:26.037564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40649 (local address 127.0.97.62:32833)
0621 02:39:26.037761 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:26.037768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:26.038243 (+ 475us) server_negotiation.cc:407] Connection header received
0621 02:39:26.038467 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:26.038471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:26.038534 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:26.038674 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:26.039447 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.040186 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.041422 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.041861 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.042001 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:26.045679 (+ 3678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40649: BlockingRecv error: recv got EOF from 127.0.97.2:40649 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 02:39:26 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:26 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:26.487303 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:26.478926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:40253)
0621 02:39:26.479452 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:26.479468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:26.479576 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:26.479921 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:26.479973 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:26.480453 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:26.480639 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:26.480648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.481826 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.481831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:26.482704 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:26.482714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.483434 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.483440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:26.483614 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:26.484291 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:26.484311 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:26.487077 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":304,"threads_started":1}
W20260621 02:39:26.487520 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:26.479072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40253 (local address 127.0.97.62:32833)
0621 02:39:26.479252 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:26.479258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:26.479545 (+ 287us) server_negotiation.cc:407] Connection header received
0621 02:39:26.479674 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:26.479678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:26.479734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:26.479837 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:26.480800 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.481665 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.482868 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.483267 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.483411 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:26.487278 (+ 3867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40253: BlockingRecv error: recv got EOF from 127.0.97.3:40253 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 02:39:26 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:26 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:26.614748 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:26.606595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:40785)
0621 02:39:26.607094 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:26.607111 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:26.607233 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:26.607521 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:26.607524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:26.607878 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:26.608025 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:26.608032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.608955 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.608958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:26.609895 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:26.609906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.610623 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.610628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:26.610851 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:26.611524 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:26.611541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:26.614523 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":271,"threads_started":1}
W20260621 02:39:26.614940 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:26.606683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40785 (local address 127.0.97.62:32833)
0621 02:39:26.606953 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:26.606959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:26.607192 (+ 233us) server_negotiation.cc:407] Connection header received
0621 02:39:26.607331 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:26.607335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:26.607402 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:26.607482 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:26.608145 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.608828 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.610006 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:26.610460 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:26.610578 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:26.614699 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40785: BlockingRecv error: recv got EOF from 127.0.97.1:40785 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Jun 21 02:39:27 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:27 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:27.055393 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:27.046728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:44405)
0621 02:39:27.047267 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:27.047290 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:27.047407 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:27.047677 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:27.047680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:27.048031 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:27.048180 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:27.048188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.049190 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.049195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:27.050324 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:27.050337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.051157 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.051161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:27.051369 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:27.052111 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:27.052129 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:27.055153 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":300,"threads_started":1}
W20260621 02:39:27.055649 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:27.046811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44405 (local address 127.0.97.62:32833)
0621 02:39:27.046995 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:27.047000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:27.047360 (+ 360us) server_negotiation.cc:407] Connection header received
0621 02:39:27.047499 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:27.047503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:27.047558 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:27.047638 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:27.048337 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.049036 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.050464 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.051008 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.051106 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:27.055415 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44405: BlockingRecv error: recv got EOF from 127.0.97.2:44405 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 02:39:27 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:27 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:27.497011 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:27.488240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:44923)
0621 02:39:27.488777 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:27.488792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:27.488909 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:27.489269 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:27.489273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:27.489769 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:27.489921 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:27.489929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.491070 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.491074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:27.491962 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:27.491973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.492702 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.492707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:27.492899 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:27.493622 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:27.493639 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:27.496711 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":298,"threads_started":1}
W20260621 02:39:27.497218 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:27.488344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44923 (local address 127.0.97.62:32833)
0621 02:39:27.488507 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:27.488512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:27.488903 (+ 391us) server_negotiation.cc:407] Connection header received
0621 02:39:27.489006 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:27.489009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:27.489068 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:27.489206 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:27.490066 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.490933 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.492102 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.492584 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.492676 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:27.496973 (+ 4297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44923: BlockingRecv error: recv got EOF from 127.0.97.3:44923 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 02:39:27 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:27 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:27.627368 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:27.615755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:42489)
0621 02:39:27.616331 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:27.616354 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:27.616518 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:27.616814 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:27.616817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:27.617180 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:27.617447 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:27.617458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.618407 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.618415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:27.619354 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:27.619365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.620168 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.620173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:27.620290 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:27.624062 (+ 3772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:27.624080 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:27.627057 (+ 2977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":431,"threads_started":1}
W20260621 02:39:27.627501 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:27.615825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42489 (local address 127.0.97.62:32833)
0621 02:39:27.616048 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:27.616055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:27.616480 (+ 425us) server_negotiation.cc:407] Connection header received
0621 02:39:27.616629 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:27.616633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:27.616691 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:27.616767 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:27.617570 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.618254 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.619494 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:27.619990 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:27.620081 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:27.627240 (+ 7159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42489: BlockingRecv error: recv got EOF from 127.0.97.1:42489 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 02:39:28 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:28 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:28.064702 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:28.056366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:57475)
0621 02:39:28.056912 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:28.056931 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:28.057043 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:28.057400 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:28.057404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:28.057790 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:28.057980 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:28.057989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.059173 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.059178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:28.060281 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:28.060293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.061028 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.061033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:28.061156 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:28.061865 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:28.061883 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:28.064476 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":280,"threads_started":1}
W20260621 02:39:28.064914 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:28.056564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57475 (local address 127.0.97.62:32833)
0621 02:39:28.056727 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:28.056731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:28.057036 (+ 305us) server_negotiation.cc:407] Connection header received
0621 02:39:28.057145 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:28.057149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:28.057208 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:28.057369 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:28.058138 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.059015 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.060414 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.060833 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.060933 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:28.064671 (+ 3738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57475: BlockingRecv error: recv got EOF from 127.0.97.2:57475 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 02:39:28 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:28 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:28.506532 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:28.498107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:53523)
0621 02:39:28.498648 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:28.498664 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:28.498778 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:28.499060 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:28.499063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:28.499448 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:28.499617 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:28.499625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.500633 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.500636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:28.501767 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:28.501782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.502735 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.502742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:28.502909 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:28.503543 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:28.503559 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:28.506303 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":291,"threads_started":1}
W20260621 02:39:28.506696 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:28.498189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53523 (local address 127.0.97.62:32833)
0621 02:39:28.498353 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:28.498358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:28.498786 (+ 428us) server_negotiation.cc:407] Connection header received
0621 02:39:28.498872 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:28.498876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:28.498935 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:28.499021 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:28.499774 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.500516 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.501928 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.502535 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.502661 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:28.506487 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53523: BlockingRecv error: recv got EOF from 127.0.97.3:53523 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 02:39:28 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:28 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:28.637531 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:28.628433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:58093)
0621 02:39:28.628985 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:28.629001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:28.629155 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:28.629554 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:28.629560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:28.629959 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:28.630108 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:28.630117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.631121 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.631127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:28.632038 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:28.632049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.632998 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.633005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:28.633218 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:28.634034 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:28.634063 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:28.637208 (+ 3145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":293,"threads_started":1}
W20260621 02:39:28.637667 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:28.628516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58093 (local address 127.0.97.62:32833)
0621 02:39:28.628853 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:28.628859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:28.629091 (+ 232us) server_negotiation.cc:407] Connection header received
0621 02:39:28.629319 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:28.629323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:28.629411 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:28.629521 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:28.630249 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.630976 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.632217 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:28.632773 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:28.632917 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:28.637444 (+ 4527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58093: BlockingRecv error: recv got EOF from 127.0.97.1:58093 (error 108)
Metrics: {"server-negotiator.queue_time_us":202}
Jun 21 02:39:29 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:29 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:29.074054 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:29.065656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:35703)
0621 02:39:29.066144 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:29.066165 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:29.066275 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:29.066568 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:29.066570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:29.066921 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:29.067064 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:29.067072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.068089 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.068094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:29.069075 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:29.069087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.069996 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.070000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:29.070193 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:29.071033 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:29.071054 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:29.073811 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":270,"threads_started":1}
W20260621 02:39:29.074230 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:29.065770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35703 (local address 127.0.97.62:32833)
0621 02:39:29.065940 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:29.065945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:29.066238 (+ 293us) server_negotiation.cc:407] Connection header received
0621 02:39:29.066376 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:29.066379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:29.066441 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:29.066519 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:29.067215 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.067947 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.069220 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.069847 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.069975 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:29.074003 (+ 4028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35703: BlockingRecv error: recv got EOF from 127.0.97.2:35703 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 02:39:29 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:29 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:29.515326 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:29.507611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.3:35231)
0621 02:39:29.508020 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:29.508037 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:29.508147 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:29.508408 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:29.508411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:29.508761 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:29.508903 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:29.508911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.509949 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.509953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:29.510858 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:29.510869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.511510 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.511515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:29.511639 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:29.512218 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:29.512234 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:29.514962 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":228,"threads_started":1}
W20260621 02:39:29.515414 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:29.507656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35231 (local address 127.0.97.62:32833)
0621 02:39:29.507900 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:29.507908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:29.508119 (+ 211us) server_negotiation.cc:407] Connection header received
0621 02:39:29.508226 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:29.508229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:29.508288 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:29.508366 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:29.509014 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.509825 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.510983 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.511356 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.511445 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:29.515192 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35231: BlockingRecv error: recv got EOF from 127.0.97.3:35231 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 02:39:29 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:29 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:29.646886 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:29.638926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.1:36863)
0621 02:39:29.639434 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:29.639451 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:29.639559 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:29.639882 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:29.639885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:29.640262 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:29.640435 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:29.640443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.641486 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.641490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:29.642434 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:29.642445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.643168 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.643172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:29.643283 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:29.643863 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:29.643882 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:29.646639 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":281,"threads_started":1}
W20260621 02:39:29.647074 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:29.638996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36863 (local address 127.0.97.62:32833)
0621 02:39:29.639215 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:29.639225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:29.639528 (+ 303us) server_negotiation.cc:407] Connection header received
0621 02:39:29.639652 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:29.639655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:29.639743 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:29.639840 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:29.640590 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.641349 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.642581 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:29.643028 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:29.643131 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:29.646842 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36863: BlockingRecv error: recv got EOF from 127.0.97.1:36863 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 02:39:30 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:39:30 dist-test-slave-jkn7 krb5kdc[8611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:39:30.083190 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:30.075166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32833 (local address 127.0.97.2:57477)
0621 02:39:30.075616 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:30.075641 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:30.075749 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:30.076025 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:30.076028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:30.076534 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:30.076701 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:30.076709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.077949 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.077955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:30.078977 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:30.078988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.079643 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.079646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:30.079752 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:30.080325 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:30.080342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:30.082940 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:32833: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":248,"threads_started":1}
W20260621 02:39:30.083377 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:30.075237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57477 (local address 127.0.97.62:32833)
0621 02:39:30.075467 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:30.075474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:30.075746 (+ 272us) server_negotiation.cc:407] Connection header received
0621 02:39:30.075842 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:30.075846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:30.075906 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:30.075978 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:30.076833 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.077773 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.079102 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.079526 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.079617 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:30.083140 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57477: BlockingRecv error: recv got EOF from 127.0.97.2:57477 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:39:30.360240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9198
I20260621 02:39:30.371939 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:30.361202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.0.97.1:40383)
0621 02:39:30.361839 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:30.361846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:30.361872 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:39:30.361926 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:30.361932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:30.362167 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:30.362328 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:30.363699 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.364713 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.366073 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.366606 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.366710 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:30.369038 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:30.369046 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:30.369058 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:30.369092 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:30.371492 (+ 2400us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 02:39:30.371717 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44254: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":350,"threads_started":1}
W20260621 02:39:30.372136 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9198: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:40383: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:39:30.372336 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 9198
I20260621 02:39:30.388463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9349
I20260621 02:39:30.399899 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:30.389374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.0.97.2:44227)
0621 02:39:30.389900 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:30.389907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:30.389945 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:39:30.390004 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:30.390011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:30.390213 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:30.390361 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:30.391526 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.392555 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.393932 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.394497 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.394612 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:30.396897 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:30.396905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:30.396918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:30.396961 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:30.399397 (+ 2436us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 02:39:30.399628 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54726: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":251,"threads_started":1}
W20260621 02:39:30.400009 9659 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54726: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 02:39:30.400043 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9349: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:44227: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:39:30.400198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 9349
I20260621 02:39:30.416340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9480
I20260621 02:39:30.428522 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:30.417343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.0.97.3:32929)
0621 02:39:30.417967 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:30.417974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:30.417999 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:39:30.418059 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:30.418065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:30.418291 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:30.418507 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:30.419820 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.420885 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.422245 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:30.423237 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:30.423350 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:30.425773 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:30.425780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:30.425793 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:30.425838 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:30.428052 (+ 2214us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 02:39:30.428284 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38650: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":328,"threads_started":1}
W20260621 02:39:30.428638 9660 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38650: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 02:39:30.428725 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9480: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:32929: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:39:30.428884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 9480
I20260621 02:39:30.444828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9129
I20260621 02:39:30.445968 9191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:30.558032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 9129
2026-06-21T02:39:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20983 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T02:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:39:30Z Disabled control of system clock
I20260621 02:39:30.631431 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46177
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 02:39:30.946225 9665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:30.946489 9665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:30.946625 9665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:30.953619 9665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:39:30.953715 9665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:30.953787 9665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:30.953907 9665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:39:30.953986 9665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:39:30.962826 9665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:30.964033 9665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:30.965709 9665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:30.977334 9671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:30.977293 9673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:30.977401 9670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:30.977522 9665 server_base.cc:1034] running on GCE node
I20260621 02:39:30.978601 9665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:30.979903 9665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:30.981127 9665 hybrid_clock.cc:648] HybridClock initialized: now 1782009570981079 us; error 41 us; skew 500 ppm
I20260621 02:39:30.981513 9665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:39:30.983747 9665 webserver.cc:460] Webserver started at http://127.0.97.62:37043/ using document root <none> and password file <none>
I20260621 02:39:30.984421 9665 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:30.984560 9665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:30.984876 9665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:30.987627 9665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "c58dab24dd894cafa8e8df0c5d074650"
format_stamp: "Formatted at 2026-06-21 02:39:30 on dist-test-slave-jkn7"
server_key: "73927f1023abece4fb893662b2181681"
server_key_iv: "6453ca57e35bd5658f73d8c15f03bbda"
server_key_version: "encryptionkey@0"
I20260621 02:39:30.988386 9665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "c58dab24dd894cafa8e8df0c5d074650"
format_stamp: "Formatted at 2026-06-21 02:39:30 on dist-test-slave-jkn7"
server_key: "73927f1023abece4fb893662b2181681"
server_key_iv: "6453ca57e35bd5658f73d8c15f03bbda"
server_key_version: "encryptionkey@0"
I20260621 02:39:30.993126 9665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260621 02:39:30.996347 9679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:30.997617 9665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:39:30.997786 9665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "c58dab24dd894cafa8e8df0c5d074650"
format_stamp: "Formatted at 2026-06-21 02:39:30 on dist-test-slave-jkn7"
server_key: "73927f1023abece4fb893662b2181681"
server_key_iv: "6453ca57e35bd5658f73d8c15f03bbda"
server_key_version: "encryptionkey@0"
I20260621 02:39:30.997999 9665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:31.011726 9665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:31.012558 9665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:31.012861 9665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:31.030435 9665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46177
I20260621 02:39:31.030459 9730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46177 every 8 connection(s)
I20260621 02:39:31.031716 9665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:39:31.035284 9731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:31.040354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 9665
I20260621 02:39:31.040531 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:39:31.041332 388 external_mini_cluster.cc:1428] Setting key 59b8553a0981c6ced1a31c4898323cab
I20260621 02:39:31.046360 9731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650: Bootstrap starting.
I20260621 02:39:31.049381 9731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:31.050340 9731 log.cc:824] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:31.053396 9731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650: No bootstrap required, opened a new log
I20260621 02:39:31.054543 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:31.043401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.0.97.62:46177)
0621 02:39:31.044406 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:31.044417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:31.044459 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:39:31.046102 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:31.046123 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:31.046607 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:31.046975 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:31.047554 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:31.048788 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:31.050205 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:31.051056 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:31.051273 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:31.052348 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:31.052355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:31.052368 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:31.052876 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:31.052948 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:31.052967 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:31.053056 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:31.053383 (+ 327us) server_negotiation.cc:299] Negotiation successful
0621 02:39:31.053691 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":378,"threads_started":1}
I20260621 02:39:31.060482 9731 raft_consensus.cc:348] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58dab24dd894cafa8e8df0c5d074650" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } }
I20260621 02:39:31.060823 9731 raft_consensus.cc:374] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:31.060928 9731 raft_consensus.cc:729] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58dab24dd894cafa8e8df0c5d074650, State: Initialized, Role: FOLLOWER
I20260621 02:39:31.061532 9731 consensus_queue.cc:260] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58dab24dd894cafa8e8df0c5d074650" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } }
I20260621 02:39:31.061777 9731 raft_consensus.cc:388] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:39:31.061910 9731 raft_consensus.cc:482] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:39:31.062083 9731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:31.064077 9731 raft_consensus.cc:504] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58dab24dd894cafa8e8df0c5d074650" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } }
I20260621 02:39:31.064538 9731 leader_election.cc:302] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c58dab24dd894cafa8e8df0c5d074650; no voters:
I20260621 02:39:31.065054 9731 leader_election.cc:288] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:39:31.065203 9736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:31.066013 9736 raft_consensus.cc:686] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [term 1 LEADER]: Becoming Leader. State: Replica: c58dab24dd894cafa8e8df0c5d074650, State: Running, Role: LEADER
I20260621 02:39:31.066480 9736 consensus_queue.cc:237] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58dab24dd894cafa8e8df0c5d074650" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } }
I20260621 02:39:31.067106 9731 sys_catalog.cc:564] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:39:31.069579 9738 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c58dab24dd894cafa8e8df0c5d074650. Latest consensus state: current_term: 1 leader_uuid: "c58dab24dd894cafa8e8df0c5d074650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58dab24dd894cafa8e8df0c5d074650" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } }
I20260621 02:39:31.069617 9737 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c58dab24dd894cafa8e8df0c5d074650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58dab24dd894cafa8e8df0c5d074650" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } }
I20260621 02:39:31.069929 9738 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:31.070013 9737 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:31.071748 9741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:39:31.076651 9741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:39:31.085116 9741 catalog_manager.cc:1347] Generated new cluster ID: 31b19ba897d3440bb4863a74a55c4f0b
I20260621 02:39:31.085218 9741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:39:31.099086 9741 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:39:31.100777 9741 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:39:31.109966 9741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c58dab24dd894cafa8e8df0c5d074650: Generated new TSK 0
I20260621 02:39:31.110925 9741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:39:31.123701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46177
--builtin_ntp_servers=127.0.97.20:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 02:39:31.448431 9755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:31.448750 9755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:31.448907 9755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:31.455955 9755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:31.456085 9755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:31.456249 9755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:39:31.465057 9755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:31.466393 9755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:31.468474 9755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:31.482017 9760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:31.481938 9763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:31.481982 9761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:31.482817 9755 server_base.cc:1034] running on GCE node
I20260621 02:39:31.483335 9755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:31.484234 9755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:31.485388 9755 hybrid_clock.cc:648] HybridClock initialized: now 1782009571485344 us; error 28 us; skew 500 ppm
I20260621 02:39:31.485652 9755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:39:31.487800 9755 webserver.cc:460] Webserver started at http://127.0.97.1:45293/ using document root <none> and password file <none>
I20260621 02:39:31.488420 9755 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:31.488516 9755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:31.488763 9755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:31.491604 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "ba5f326d8ff64673ac7e057a1f811e6d"
format_stamp: "Formatted at 2026-06-21 02:39:31 on dist-test-slave-jkn7"
server_key: "b5ebc356f07e8bb91d9391b0dc0955e4"
server_key_iv: "55a5d1a945df4cf6bbe0c9e159d2465a"
server_key_version: "encryptionkey@0"
I20260621 02:39:31.492316 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ba5f326d8ff64673ac7e057a1f811e6d"
format_stamp: "Formatted at 2026-06-21 02:39:31 on dist-test-slave-jkn7"
server_key: "b5ebc356f07e8bb91d9391b0dc0955e4"
server_key_iv: "55a5d1a945df4cf6bbe0c9e159d2465a"
server_key_version: "encryptionkey@0"
I20260621 02:39:31.497648 9755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:39:31.501648 9769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:31.503145 9755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:39:31.503327 9755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "ba5f326d8ff64673ac7e057a1f811e6d"
format_stamp: "Formatted at 2026-06-21 02:39:31 on dist-test-slave-jkn7"
server_key: "b5ebc356f07e8bb91d9391b0dc0955e4"
server_key_iv: "55a5d1a945df4cf6bbe0c9e159d2465a"
server_key_version: "encryptionkey@0"
I20260621 02:39:31.503564 9755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:31.531790 9755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:31.532666 9755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:31.532960 9755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:31.533936 9755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:31.535362 9755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:31.535488 9755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:31.535619 9755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:31.535724 9755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:31.567744 9755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35401
I20260621 02:39:31.567839 9881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35401 every 8 connection(s)
I20260621 02:39:31.569058 9755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:39:31.574543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 9755
I20260621 02:39:31.574750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:39:31.575541 388 external_mini_cluster.cc:1428] Setting key 9fc1e97cda54a19337b9bb9af6237fce
I20260621 02:39:31.578166 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46177
--builtin_ntp_servers=127.0.97.20:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 02:39:31.582273 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:31.571595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47315 (local address 127.0.97.62:46177)
0621 02:39:31.572025 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:31.572049 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:31.573356 (+ 1307us) server_negotiation.cc:407] Connection header received
0621 02:39:31.575153 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:31.575162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:31.575254 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:31.575404 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:31.576492 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:31.577401 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:31.578987 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:31.579735 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:31.579889 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:31.581183 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:31.581192 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:31.581199 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:31.581224 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:31.581275 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:31.581316 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:31.581417 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:31.581714 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 02:39:31.581843 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":232,"threads_started":1}
I20260621 02:39:31.583158 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:31.572031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46177 (local address 127.0.97.1:47315)
0621 02:39:31.573125 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:31.573169 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:31.574893 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:31.575659 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:31.575672 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:31.575787 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:31.576337 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:31.576354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:31.577559 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:31.577563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:31.578843 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:31.578856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:31.579955 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:31.579960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:31.580152 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:31.580873 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:39:31.580876 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:31.580916 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:39:31.580996 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:31.581491 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:31.581500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:31.581507 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:39:31.581682 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 02:39:31.582013 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":349,"threads_started":1}
I20260621 02:39:31.584748 9882 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46177
I20260621 02:39:31.585090 9882 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:31.585902 9882 heartbeater.cc:511] Master 127.0.97.62:46177 requested a full tablet report, sending...
I20260621 02:39:31.587976 9696 ts_manager.cc:194] Registered new tserver with Master: ba5f326d8ff64673ac7e057a1f811e6d (127.0.97.1:35401)
I20260621 02:39:31.589519 9696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:47315
W20260621 02:39:31.900033 9886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:31.900300 9886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:31.900437 9886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:31.907795 9886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:31.907929 9886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:31.908100 9886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:39:31.918200 9886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:31.919449 9886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:31.921201 9886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:31.933930 9891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:31.933982 9894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:31.934010 9892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:31.934805 9886 server_base.cc:1034] running on GCE node
I20260621 02:39:31.935377 9886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:31.936271 9886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:31.937494 9886 hybrid_clock.cc:648] HybridClock initialized: now 1782009571937448 us; error 58 us; skew 500 ppm
I20260621 02:39:31.937798 9886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:39:31.940060 9886 webserver.cc:460] Webserver started at http://127.0.97.2:34259/ using document root <none> and password file <none>
I20260621 02:39:31.940742 9886 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:31.940871 9886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:31.941205 9886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:31.944013 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "f3114926e14948f6a96a878a9db856b8"
format_stamp: "Formatted at 2026-06-21 02:39:31 on dist-test-slave-jkn7"
server_key: "42d03350e2dd99fa17fa19bc7a94802e"
server_key_iv: "134ad08b5b9dc1d71e47e8ca55b52d01"
server_key_version: "encryptionkey@0"
I20260621 02:39:31.944797 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f3114926e14948f6a96a878a9db856b8"
format_stamp: "Formatted at 2026-06-21 02:39:31 on dist-test-slave-jkn7"
server_key: "42d03350e2dd99fa17fa19bc7a94802e"
server_key_iv: "134ad08b5b9dc1d71e47e8ca55b52d01"
server_key_version: "encryptionkey@0"
I20260621 02:39:31.949859 9886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:39:31.953148 9900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:31.954530 9886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:39:31.954723 9886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "f3114926e14948f6a96a878a9db856b8"
format_stamp: "Formatted at 2026-06-21 02:39:31 on dist-test-slave-jkn7"
server_key: "42d03350e2dd99fa17fa19bc7a94802e"
server_key_iv: "134ad08b5b9dc1d71e47e8ca55b52d01"
server_key_version: "encryptionkey@0"
I20260621 02:39:31.954942 9886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:31.972702 9886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:31.973706 9886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:31.973968 9886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:31.974848 9886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:31.976226 9886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:31.976354 9886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:31.976470 9886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:31.976563 9886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:32.010061 9886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32847
I20260621 02:39:32.010114 10012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32847 every 8 connection(s)
I20260621 02:39:32.011497 9886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:39:32.012640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 9886
I20260621 02:39:32.012807 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:39:32.013383 388 external_mini_cluster.cc:1428] Setting key 68fa197ac8f7b3d03dd0339650beaa04
I20260621 02:39:32.016486 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46177
--builtin_ntp_servers=127.0.97.20:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 02:39:32.026856 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:32.014584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34389 (local address 127.0.97.62:46177)
0621 02:39:32.014806 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:32.014813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:32.016366 (+ 1553us) server_negotiation.cc:407] Connection header received
0621 02:39:32.019244 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:32.019248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:32.019316 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:32.019415 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:32.020828 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.021655 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.024163 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.024553 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.024684 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:32.026071 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:32.026078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:32.026087 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:32.026104 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:32.026133 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:32.026144 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:32.026238 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:32.026452 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 02:39:32.026552 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 02:39:32.027848 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:32.015010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46177 (local address 127.0.97.2:34389)
0621 02:39:32.016193 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:32.016240 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:32.018902 (+ 2662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:32.019680 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:32.019694 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:32.019816 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:32.020617 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:32.020642 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.021828 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.021836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:32.023984 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:32.024023 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.024788 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.024797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:32.025024 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:32.025757 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:39:32.025760 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:32.025800 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:39:32.025876 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:32.026236 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:32.026248 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:32.026258 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:39:32.026445 (+ 187us) client_negotiation.cc:240] Negotiation successful
0621 02:39:32.026828 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":373,"threads_started":1}
I20260621 02:39:32.029265 10013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46177
I20260621 02:39:32.029608 10013 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:32.030537 10013 heartbeater.cc:511] Master 127.0.97.62:46177 requested a full tablet report, sending...
I20260621 02:39:32.032096 9696 ts_manager.cc:194] Registered new tserver with Master: f3114926e14948f6a96a878a9db856b8 (127.0.97.2:32847)
I20260621 02:39:32.033427 9696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:34389
W20260621 02:39:32.336575 10016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:32.336885 10016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:32.337044 10016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:32.344188 10016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:32.344312 10016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:32.344502 10016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:39:32.353372 10016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:32.354554 10016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:32.356246 10016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:32.370134 10021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:32.370203 10024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:32.370152 10022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:32.371300 10016 server_base.cc:1034] running on GCE node
I20260621 02:39:32.371937 10016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:32.372978 10016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:32.374243 10016 hybrid_clock.cc:648] HybridClock initialized: now 1782009572374181 us; error 58 us; skew 500 ppm
I20260621 02:39:32.374588 10016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:39:32.376801 10016 webserver.cc:460] Webserver started at http://127.0.97.3:41773/ using document root <none> and password file <none>
I20260621 02:39:32.377588 10016 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:32.377722 10016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:32.378054 10016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:32.380872 10016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "630bc5a0b23e4d3f8a7eed21987378e1"
format_stamp: "Formatted at 2026-06-21 02:39:32 on dist-test-slave-jkn7"
server_key: "5ed944ae52d838a15d93b43f0ee6cc61"
server_key_iv: "bc180703ed75fbd77dd58bc5445083df"
server_key_version: "encryptionkey@0"
I20260621 02:39:32.381693 10016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "630bc5a0b23e4d3f8a7eed21987378e1"
format_stamp: "Formatted at 2026-06-21 02:39:32 on dist-test-slave-jkn7"
server_key: "5ed944ae52d838a15d93b43f0ee6cc61"
server_key_iv: "bc180703ed75fbd77dd58bc5445083df"
server_key_version: "encryptionkey@0"
I20260621 02:39:32.386523 10016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:39:32.389705 10030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:32.390925 10016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260621 02:39:32.391091 10016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "630bc5a0b23e4d3f8a7eed21987378e1"
format_stamp: "Formatted at 2026-06-21 02:39:32 on dist-test-slave-jkn7"
server_key: "5ed944ae52d838a15d93b43f0ee6cc61"
server_key_iv: "bc180703ed75fbd77dd58bc5445083df"
server_key_version: "encryptionkey@0"
I20260621 02:39:32.391292 10016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:32.410442 10016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:32.411178 10016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:32.411441 10016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:32.412321 10016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:32.413722 10016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:32.413843 10016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:32.413985 10016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:32.414076 10016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:32.446489 10016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33159
I20260621 02:39:32.446520 10142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33159 every 8 connection(s)
I20260621 02:39:32.447803 10016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:39:32.450702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 10016
I20260621 02:39:32.450877 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:39:32.451345 388 external_mini_cluster.cc:1428] Setting key 74f36e8478f2128b77b99e1524cce64b
I20260621 02:39:32.461613 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:32.450534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54565 (local address 127.0.97.62:46177)
0621 02:39:32.450745 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:32.450751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:32.452383 (+ 1632us) server_negotiation.cc:407] Connection header received
0621 02:39:32.455196 (+ 2813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:32.455201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:32.455284 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:32.455399 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:32.456453 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.457190 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.458835 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.459253 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.459420 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:32.460747 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:32.460753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:32.460758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:32.460785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:32.460821 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:32.460833 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:32.460927 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:32.461134 (+ 207us) server_negotiation.cc:299] Negotiation successful
0621 02:39:32.461225 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:39:32.462556 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:32.451145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46177 (local address 127.0.97.3:54565)
0621 02:39:32.452228 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:32.452282 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:32.454944 (+ 2662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:32.455633 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:32.455642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:32.455730 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:32.456289 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:32.456314 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.457375 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.457379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:32.458702 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:32.458714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.459496 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.459505 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:32.459727 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:32.460438 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:39:32.460440 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:32.460477 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:39:32.460554 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:32.460925 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:32.460937 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:32.460947 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:39:32.461126 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 02:39:32.461544 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":354,"threads_started":1}
I20260621 02:39:32.463829 10143 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46177
I20260621 02:39:32.464121 10143 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:32.464857 10143 heartbeater.cc:511] Master 127.0.97.62:46177 requested a full tablet report, sending...
I20260621 02:39:32.466383 9696 ts_manager.cc:194] Registered new tserver with Master: 630bc5a0b23e4d3f8a7eed21987378e1 (127.0.97.3:33159)
I20260621 02:39:32.467056 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:39:32.467516 9696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:54565
W20260621 02:39:32.473208 9884 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:32.471759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.0.97.62:46177)
0621 02:39:32.471920 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:32.471924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:32.472393 (+ 469us) server_negotiation.cc:407] Connection header received
0621 02:39:32.472539 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:32.472542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:32.472611 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:32.472685 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:32.472975 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50190: BlockingRecv error: recv got EOF from 127.0.0.1:50190 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:39:32.473798 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9755
I20260621 02:39:32.480983 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:32.474696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.0.97.1:35401)
0621 02:39:32.475181 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:32.475194 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:32.475219 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:39:32.475272 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:32.475277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:32.475459 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:32.475579 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:32.475895 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.476839 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.478113 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.478641 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.478740 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:32.479691 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:32.479698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:32.479712 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:32.480280 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:32.480327 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:32.480346 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:32.480433 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:32.480565 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 02:39:32.480672 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":247,"threads_started":1}
I20260621 02:39:32.482254 9877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:32.562610 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 9755
I20260621 02:39:32.580044 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9886
I20260621 02:39:32.587407 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:32.580874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.0.97.2:32847)
0621 02:39:32.581398 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:32.581406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:32.581435 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:39:32.581498 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:32.581505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:32.581711 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:32.581859 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:32.582214 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.583145 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.584351 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.584891 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.584994 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:32.586032 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:32.586039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:32.586053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:32.586611 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:32.586653 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:32.586675 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:32.586762 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:32.586921 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 02:39:32.587069 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":255,"threads_started":1}
I20260621 02:39:32.588596 10008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:32.668974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 9886
I20260621 02:39:32.685148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 10016
I20260621 02:39:32.696321 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:32.686194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.0.97.3:33159)
0621 02:39:32.686844 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:32.686851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:32.686884 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:39:32.686943 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:32.686949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:32.687174 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:32.687316 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:32.687716 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.688721 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.690144 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:32.690806 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:32.690912 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:32.694941 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:32.694951 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:32.694965 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:32.695543 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:32.695590 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:32.695609 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:32.695702 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:32.695842 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 02:39:32.696017 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":346,"threads_started":1}
I20260621 02:39:32.697536 10138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:32.778641 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 10016
I20260621 02:39:32.795133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 9665
I20260621 02:39:32.796327 9726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:32.895921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 9665
2026-06-21T02:39:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2332 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T02:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:39:32Z Disabled control of system clock
I20260621 02:39:32.962953 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37367
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 02:39:33.274355 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:33.274613 10162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:33.274789 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:33.282040 10162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:39:33.282128 10162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:33.282214 10162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:33.282312 10162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:39:33.282394 10162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:39:33.291090 10162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:33.292224 10162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:33.293964 10162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:33.305101 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:33.305059 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:33.305197 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:33.305935 10162 server_base.cc:1034] running on GCE node
I20260621 02:39:33.306700 10162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:33.307958 10162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:33.309185 10162 hybrid_clock.cc:648] HybridClock initialized: now 1782009573309141 us; error 53 us; skew 500 ppm
I20260621 02:39:33.309527 10162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:39:33.311638 10162 webserver.cc:460] Webserver started at http://127.0.97.62:40407/ using document root <none> and password file <none>
I20260621 02:39:33.312281 10162 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:33.312407 10162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:33.312708 10162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:33.315459 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "a5c062b54a2541aab3ed0146824096f2"
format_stamp: "Formatted at 2026-06-21 02:39:33 on dist-test-slave-jkn7"
server_key: "8fa9880dd5225ac46e90bd24d89c5fdd"
server_key_iv: "978c82715ea8e066a5ac5e9e58a77d5d"
server_key_version: "encryptionkey@0"
I20260621 02:39:33.316211 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "a5c062b54a2541aab3ed0146824096f2"
format_stamp: "Formatted at 2026-06-21 02:39:33 on dist-test-slave-jkn7"
server_key: "8fa9880dd5225ac46e90bd24d89c5fdd"
server_key_iv: "978c82715ea8e066a5ac5e9e58a77d5d"
server_key_version: "encryptionkey@0"
I20260621 02:39:33.320871 10162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260621 02:39:33.324052 10176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:33.325264 10162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:39:33.325503 10162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "a5c062b54a2541aab3ed0146824096f2"
format_stamp: "Formatted at 2026-06-21 02:39:33 on dist-test-slave-jkn7"
server_key: "8fa9880dd5225ac46e90bd24d89c5fdd"
server_key_iv: "978c82715ea8e066a5ac5e9e58a77d5d"
server_key_version: "encryptionkey@0"
I20260621 02:39:33.325713 10162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:33.331018 10162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:33.331707 10162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:33.331996 10162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:33.349222 10162 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37367
I20260621 02:39:33.349244 10227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37367 every 8 connection(s)
I20260621 02:39:33.350469 10162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:39:33.354071 10228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:33.359730 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 10162
I20260621 02:39:33.359964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:39:33.360554 388 external_mini_cluster.cc:1428] Setting key a583a227ff0870ee44ba970ef2b675f7
I20260621 02:39:33.367017 10228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2: Bootstrap starting.
I20260621 02:39:33.368916 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:33.362538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.0.97.62:37367)
0621 02:39:33.363593 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:33.363607 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:33.363656 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:39:33.365629 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:33.365651 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:33.366024 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:33.366323 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:33.367404 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:33.367411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:33.367423 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:33.367437 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:33.367496 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:33.367512 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:33.367680 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:33.367789 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 02:39:33.368046 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":351,"threads_started":1}
I20260621 02:39:33.370383 10228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:33.371622 10228 log.cc:824] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:33.374069 10228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2: No bootstrap required, opened a new log
I20260621 02:39:33.380909 10228 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c062b54a2541aab3ed0146824096f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37367 } }
I20260621 02:39:33.381253 10228 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:33.381417 10228 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c062b54a2541aab3ed0146824096f2, State: Initialized, Role: FOLLOWER
I20260621 02:39:33.382006 10228 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c062b54a2541aab3ed0146824096f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37367 } }
I20260621 02:39:33.382236 10228 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:39:33.382381 10228 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:39:33.382550 10228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:33.384723 10228 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c062b54a2541aab3ed0146824096f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37367 } }
I20260621 02:39:33.385177 10228 leader_election.cc:302] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5c062b54a2541aab3ed0146824096f2; no voters:
I20260621 02:39:33.385717 10228 leader_election.cc:288] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:39:33.385845 10233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:33.386521 10233 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [term 1 LEADER]: Becoming Leader. State: Replica: a5c062b54a2541aab3ed0146824096f2, State: Running, Role: LEADER
I20260621 02:39:33.387069 10233 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c062b54a2541aab3ed0146824096f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37367 } }
I20260621 02:39:33.387585 10228 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:39:33.389829 10235 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5c062b54a2541aab3ed0146824096f2. Latest consensus state: current_term: 1 leader_uuid: "a5c062b54a2541aab3ed0146824096f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c062b54a2541aab3ed0146824096f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37367 } } }
I20260621 02:39:33.390210 10235 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:33.390686 10234 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5c062b54a2541aab3ed0146824096f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c062b54a2541aab3ed0146824096f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37367 } } }
I20260621 02:39:33.391026 10234 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:33.391367 10241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:39:33.396945 10241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:39:33.405472 10241 catalog_manager.cc:1347] Generated new cluster ID: f1b0548a62ea412aac41c685c68b3bb1
I20260621 02:39:33.405612 10241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:39:33.423740 10241 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:39:33.424204 10241 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:39:33.442893 10241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5c062b54a2541aab3ed0146824096f2: Generated new TSK 0
I20260621 02:39:33.443898 10241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:39:33.457767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37367
--builtin_ntp_servers=127.0.97.20:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 02:39:33.773038 10252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:33.773372 10252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:33.773536 10252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:33.780761 10252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:33.780905 10252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:33.781070 10252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:39:33.790239 10252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:33.791546 10252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:33.793574 10252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:33.806442 10257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:33.806529 10260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:33.806571 10258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:33.807366 10252 server_base.cc:1034] running on GCE node
I20260621 02:39:33.807914 10252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:33.808812 10252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:33.810043 10252 hybrid_clock.cc:648] HybridClock initialized: now 1782009573809980 us; error 53 us; skew 500 ppm
I20260621 02:39:33.810375 10252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:39:33.812544 10252 webserver.cc:460] Webserver started at http://127.0.97.1:39357/ using document root <none> and password file <none>
I20260621 02:39:33.813202 10252 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:33.813364 10252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:33.813675 10252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:33.816365 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "f905d212ca7247cb83905aaa171fcf89"
format_stamp: "Formatted at 2026-06-21 02:39:33 on dist-test-slave-jkn7"
server_key: "93b5eea9dde9f6a2eb0d99d038e76782"
server_key_iv: "6a72eeb51262e9e1062f3ed2f99d11df"
server_key_version: "encryptionkey@0"
I20260621 02:39:33.817130 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f905d212ca7247cb83905aaa171fcf89"
format_stamp: "Formatted at 2026-06-21 02:39:33 on dist-test-slave-jkn7"
server_key: "93b5eea9dde9f6a2eb0d99d038e76782"
server_key_iv: "6a72eeb51262e9e1062f3ed2f99d11df"
server_key_version: "encryptionkey@0"
I20260621 02:39:33.822237 10252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:39:33.825601 10266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:33.826908 10252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:39:33.827095 10252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "f905d212ca7247cb83905aaa171fcf89"
format_stamp: "Formatted at 2026-06-21 02:39:33 on dist-test-slave-jkn7"
server_key: "93b5eea9dde9f6a2eb0d99d038e76782"
server_key_iv: "6a72eeb51262e9e1062f3ed2f99d11df"
server_key_version: "encryptionkey@0"
I20260621 02:39:33.827327 10252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:33.833088 10252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:33.834039 10252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:33.834304 10252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:33.835309 10252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:33.836699 10252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:33.836825 10252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:33.836951 10252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:33.837035 10252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:33.870472 10252 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33699
I20260621 02:39:33.870558 10378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33699 every 8 connection(s)
I20260621 02:39:33.872144 10252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:39:33.881381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 10252
I20260621 02:39:33.881616 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:39:33.882227 388 external_mini_cluster.cc:1428] Setting key b99fc483f7c3dc88c127b3fa12cd4da8
I20260621 02:39:33.883220 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:33.875961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37367 (local address 127.0.97.1:60507)
0621 02:39:33.877182 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:33.877237 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:33.879494 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:33.880216 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:33.880224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:33.880322 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:33.880978 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:39:33.880981 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:33.881021 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:39:33.881110 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:33.881541 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:33.881552 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:33.881559 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:39:33.881879 (+ 320us) client_negotiation.cc:240] Negotiation successful
0621 02:39:33.882105 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":343,"threads_started":1}
I20260621 02:39:33.883960 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:33.875338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60507 (local address 127.0.97.62:37367)
0621 02:39:33.875829 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:33.875836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:33.877404 (+ 1568us) server_negotiation.cc:407] Connection header received
0621 02:39:33.879836 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:33.879840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:33.879900 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:33.880008 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:33.881328 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:33.881335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:33.881340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:33.881360 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:33.881419 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:33.881432 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:33.881568 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:33.883613 (+ 2045us) server_negotiation.cc:299] Negotiation successful
0621 02:39:33.883667 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":251,"threads_started":1}
I20260621 02:39:33.885083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37367
--builtin_ntp_servers=127.0.97.20:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 02:39:33.885162 10379 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37367
I20260621 02:39:33.885502 10379 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:33.886106 10379 heartbeater.cc:511] Master 127.0.97.62:37367 requested a full tablet report, sending...
I20260621 02:39:33.888372 10193 ts_manager.cc:194] Registered new tserver with Master: f905d212ca7247cb83905aaa171fcf89 (127.0.97.1:33699)
W20260621 02:39:34.206436 10383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:34.206720 10383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:34.206866 10383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:34.214147 10383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:34.214254 10383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:34.214423 10383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:39:34.223328 10383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:34.224618 10383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:34.226313 10383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:34.238958 10388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:34.238898 10389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:34.238870 10391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:34.239871 10383 server_base.cc:1034] running on GCE node
I20260621 02:39:34.240406 10383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:34.241237 10383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:34.242470 10383 hybrid_clock.cc:648] HybridClock initialized: now 1782009574242433 us; error 28 us; skew 500 ppm
I20260621 02:39:34.242710 10383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:39:34.244630 10383 webserver.cc:460] Webserver started at http://127.0.97.2:40119/ using document root <none> and password file <none>
I20260621 02:39:34.245219 10383 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:34.245352 10383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:34.245617 10383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:34.249897 10383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "86c2b904dbd84b2ab471ddc4e339b472"
format_stamp: "Formatted at 2026-06-21 02:39:34 on dist-test-slave-jkn7"
server_key: "c714471ca56d76b7195c45f7c0ba24ac"
server_key_iv: "bfd78173862f87469771a8136466bfb5"
server_key_version: "encryptionkey@0"
I20260621 02:39:34.250567 10383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "86c2b904dbd84b2ab471ddc4e339b472"
format_stamp: "Formatted at 2026-06-21 02:39:34 on dist-test-slave-jkn7"
server_key: "c714471ca56d76b7195c45f7c0ba24ac"
server_key_iv: "bfd78173862f87469771a8136466bfb5"
server_key_version: "encryptionkey@0"
I20260621 02:39:34.255122 10383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260621 02:39:34.258741 10397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:34.259963 10383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:39:34.260130 10383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "86c2b904dbd84b2ab471ddc4e339b472"
format_stamp: "Formatted at 2026-06-21 02:39:34 on dist-test-slave-jkn7"
server_key: "c714471ca56d76b7195c45f7c0ba24ac"
server_key_iv: "bfd78173862f87469771a8136466bfb5"
server_key_version: "encryptionkey@0"
I20260621 02:39:34.260353 10383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:34.265599 10383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:34.266319 10383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:34.266577 10383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:34.267400 10383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:34.268749 10383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:34.268877 10383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:34.269001 10383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:34.269086 10383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:34.299403 10383 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34041
I20260621 02:39:34.299485 10509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34041 every 8 connection(s)
I20260621 02:39:34.300621 10383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:39:34.308171 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:34.303175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57433 (local address 127.0.97.62:37367)
0621 02:39:34.303391 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:34.303395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:34.304678 (+ 1283us) server_negotiation.cc:407] Connection header received
0621 02:39:34.306235 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:34.306239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:34.306298 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:34.306390 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:34.307608 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:34.307611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:34.307614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:34.307626 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:34.307649 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:34.307659 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:34.307741 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:34.307902 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 02:39:34.307944 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:39:34.308857 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 10383
I20260621 02:39:34.309034 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:39:34.309070 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:34.303646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37367 (local address 127.0.97.2:57433)
0621 02:39:34.304526 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:34.304559 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:34.306010 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:34.306572 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:34.306579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:34.306660 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:34.307367 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:39:34.307369 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:34.307403 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:39:34.307480 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:34.307739 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:34.307745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:34.307750 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:39:34.307962 (+ 212us) client_negotiation.cc:240] Negotiation successful
0621 02:39:34.308135 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":293,"threads_started":1}
I20260621 02:39:34.309644 388 external_mini_cluster.cc:1428] Setting key ed3e6d368f475c9d33766fddea900e86
I20260621 02:39:34.310443 10510 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37367
I20260621 02:39:34.310732 10510 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:34.311326 10510 heartbeater.cc:511] Master 127.0.97.62:37367 requested a full tablet report, sending...
I20260621 02:39:34.312366 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37367
--builtin_ntp_servers=127.0.97.20:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 02:39:34.312697 10193 ts_manager.cc:194] Registered new tserver with Master: 86c2b904dbd84b2ab471ddc4e339b472 (127.0.97.2:34041)
W20260621 02:39:34.653474 10513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:34.653728 10513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:34.653891 10513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:34.660835 10513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:34.660964 10513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:34.661144 10513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:39:34.670156 10513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:34.671377 10513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:34.673060 10513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:34.685461 10518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:34.685580 10519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:34.685691 10521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:34.686398 10513 server_base.cc:1034] running on GCE node
I20260621 02:39:34.686950 10513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:34.687877 10513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:34.689109 10513 hybrid_clock.cc:648] HybridClock initialized: now 1782009574689033 us; error 79 us; skew 500 ppm
I20260621 02:39:34.689469 10513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:39:34.691725 10513 webserver.cc:460] Webserver started at http://127.0.97.3:36411/ using document root <none> and password file <none>
I20260621 02:39:34.692517 10513 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:34.692638 10513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:34.692953 10513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:34.695740 10513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "ed85799ae4394a80ac1055f4c6960855"
format_stamp: "Formatted at 2026-06-21 02:39:34 on dist-test-slave-jkn7"
server_key: "55a616d171da8f5322bf7bb3d29bf60f"
server_key_iv: "e32cbdc680b79e5634d78c4f048545a7"
server_key_version: "encryptionkey@0"
I20260621 02:39:34.696492 10513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed85799ae4394a80ac1055f4c6960855"
format_stamp: "Formatted at 2026-06-21 02:39:34 on dist-test-slave-jkn7"
server_key: "55a616d171da8f5322bf7bb3d29bf60f"
server_key_iv: "e32cbdc680b79e5634d78c4f048545a7"
server_key_version: "encryptionkey@0"
I20260621 02:39:34.701325 10513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260621 02:39:34.704792 10527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:34.706213 10513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:39:34.706398 10513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "ed85799ae4394a80ac1055f4c6960855"
format_stamp: "Formatted at 2026-06-21 02:39:34 on dist-test-slave-jkn7"
server_key: "55a616d171da8f5322bf7bb3d29bf60f"
server_key_iv: "e32cbdc680b79e5634d78c4f048545a7"
server_key_version: "encryptionkey@0"
I20260621 02:39:34.706640 10513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:34.712038 10513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:34.712783 10513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:34.713057 10513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:34.713986 10513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:34.715281 10513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:34.715408 10513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:34.715530 10513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:34.715616 10513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:34.747447 10513 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42633
I20260621 02:39:34.747534 10639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42633 every 8 connection(s)
I20260621 02:39:34.748735 10513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:39:34.752581 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 10513
I20260621 02:39:34.752769 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:39:34.753458 388 external_mini_cluster.cc:1428] Setting key 7f8c3cfb5bf0a57908955199f8b1dc25
I20260621 02:39:34.757604 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:34.751277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46709 (local address 127.0.97.62:37367)
0621 02:39:34.751477 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:34.751482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:34.753229 (+ 1747us) server_negotiation.cc:407] Connection header received
0621 02:39:34.755468 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:34.755472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:34.755562 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:34.755673 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:34.756994 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:34.756998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:34.757002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:34.757017 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:34.757047 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:34.757058 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:34.757152 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:34.757356 (+ 204us) server_negotiation.cc:299] Negotiation successful
0621 02:39:34.757401 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:39:34.758493 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:34.751742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37367 (local address 127.0.97.3:46709)
0621 02:39:34.753049 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:34.753100 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:34.755170 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:34.755861 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:34.755869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:34.755954 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:34.756713 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:39:34.756717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:34.756762 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:39:34.756842 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:34.757143 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:34.757149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:34.757155 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:39:34.757425 (+ 270us) client_negotiation.cc:240] Negotiation successful
0621 02:39:34.757606 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":544,"threads_started":1}
I20260621 02:39:34.759687 10640 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37367
I20260621 02:39:34.759959 10640 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:34.760501 10640 heartbeater.cc:511] Master 127.0.97.62:37367 requested a full tablet report, sending...
I20260621 02:39:34.761761 10193 ts_manager.cc:194] Registered new tserver with Master: ed85799ae4394a80ac1055f4c6960855 (127.0.97.3:42633)
I20260621 02:39:34.768898 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 02:39:34.774895 10381 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:39:34.773208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.0.97.62:37367)
0621 02:39:34.773444 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:34.773449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:34.773984 (+ 535us) server_negotiation.cc:407] Connection header received
0621 02:39:34.774175 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:34.774181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:34.774239 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:34.774337 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:34.774652 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49238: BlockingRecv error: recv got EOF from 127.0.0.1:49238 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:39:34.775573 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 10252
I20260621 02:39:34.779111 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:34.776551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.0.97.1:33699)
0621 02:39:34.777057 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:34.777064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:34.777089 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:39:34.777142 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:34.777147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:34.777416 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:34.777538 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:34.778452 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:34.778458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:34.778468 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:34.778487 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:34.778528 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:34.778542 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:34.778642 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:34.778721 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 02:39:34.778815 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":260,"threads_started":1}
I20260621 02:39:34.780253 10374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:34.861143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 10252
I20260621 02:39:34.877686 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 10383
I20260621 02:39:34.881151 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:34.878597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.0.97.2:34041)
0621 02:39:34.879067 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:34.879074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:34.879100 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:39:34.879160 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:34.879166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:34.879378 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:34.879514 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:34.880481 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:34.880488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:34.880498 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:34.880511 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:34.880554 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:34.880572 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:34.880672 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:34.880761 (+ 89us) server_negotiation.cc:299] Negotiation successful
0621 02:39:34.880877 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260621 02:39:34.882300 10505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:34.963290 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 10383
I20260621 02:39:34.979516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 10513
I20260621 02:39:34.983546 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:34.980532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.0.97.3:42633)
0621 02:39:34.981109 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:34.981116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:34.981144 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:39:34.981203 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:34.981210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:34.981506 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:34.981670 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:34.982815 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:34.982820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:34.982832 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:39:34.982847 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:34.982884 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:39:34.982897 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:34.983089 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:34.983160 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 02:39:34.983253 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":283,"threads_started":1}
I20260621 02:39:34.984751 10635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:35.064920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 10513
I20260621 02:39:35.081039 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 10162
I20260621 02:39:35.082247 10223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:35.182493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 10162
2026-06-21T02:39:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2286 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:39:35 dist-test-slave-jkn7 krb5kdc[10658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:39:35 dist-test-slave-jkn7 krb5kdc[10658](info): set up 2 sockets
Jun 21 02:39:35 dist-test-slave-jkn7 krb5kdc[10658](info): commencing operation
krb5kdc: starting...
W20260621 02:39:36.274076 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:39:36 dist-test-slave-jkn7 krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:39:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:39:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:39:36.484663 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40077
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:36.792014 10671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:36.792382 10671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:36.792559 10671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:36.799924 10671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:39:36.800102 10671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:36.800220 10671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:36.800321 10671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:39:36.800396 10671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:39:36.810849 10671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:36.812273 10671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:36.813642 10671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:36.825114 10676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:36.825019 10679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:36.825062 10677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:36.825853 10671 server_base.cc:1034] running on GCE node
I20260621 02:39:36.826676 10671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:36.828153 10671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:36.829391 10671 hybrid_clock.cc:648] HybridClock initialized: now 1782009576829321 us; error 52 us; skew 500 ppm
I20260621 02:39:36.829741 10671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:36 dist-test-slave-jkn7 krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:36.832787 10671 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:39:36.834359 10671 webserver.cc:460] Webserver started at http://127.0.97.62:42429/ using document root <none> and password file <none>
I20260621 02:39:36.835059 10671 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:36.835201 10671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:36.835501 10671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:36.839057 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "0ea8f36f875d44e3bc1b6ee12197e647"
format_stamp: "Formatted at 2026-06-21 02:39:36 on dist-test-slave-jkn7"
server_key: "6c8f5788375e119e4def8a6afeb52366"
server_key_iv: "54581efc289a0a537db4675bf2ff69eb"
server_key_version: "encryptionkey@0"
I20260621 02:39:36.839839 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "0ea8f36f875d44e3bc1b6ee12197e647"
format_stamp: "Formatted at 2026-06-21 02:39:36 on dist-test-slave-jkn7"
server_key: "6c8f5788375e119e4def8a6afeb52366"
server_key_iv: "54581efc289a0a537db4675bf2ff69eb"
server_key_version: "encryptionkey@0"
I20260621 02:39:36.845118 10671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:39:36.848479 10686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:36.849879 10671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:39:36.850060 10671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "0ea8f36f875d44e3bc1b6ee12197e647"
format_stamp: "Formatted at 2026-06-21 02:39:36 on dist-test-slave-jkn7"
server_key: "6c8f5788375e119e4def8a6afeb52366"
server_key_iv: "54581efc289a0a537db4675bf2ff69eb"
server_key_version: "encryptionkey@0"
I20260621 02:39:36.850255 10671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:36.865988 10671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:36.869937 10671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:36.870203 10671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:36.888180 10671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40077
I20260621 02:39:36.888209 10737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40077 every 8 connection(s)
I20260621 02:39:36.889583 10671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:39:36.893136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 10671
I20260621 02:39:36.893375 10738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:36.893384 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:39:36.894178 388 external_mini_cluster.cc:1428] Setting key 46a57da21d743bb467c5a040d49f094c
I20260621 02:39:36.907407 10738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647: Bootstrap starting.
Jun 21 02:39:36 dist-test-slave-jkn7 krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:39:36.910964 10738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:36.911989 10738 log.cc:824] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:36.914752 10738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647: No bootstrap required, opened a new log
I20260621 02:39:36.917454 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:36.896217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.0.97.62:40077)
0621 02:39:36.897066 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:36.897078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:36.897118 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:39:36.899456 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:36.899480 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:36.899873 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:36.900208 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:36.902001 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:36.903246 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:36.905084 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:36.905841 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:36.905988 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:36.910562 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:36.910570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:36.910584 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:36.910617 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:36.913583 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:36.914245 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:36.914259 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:36.914265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:36.914433 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:36.914792 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:36.914798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:36.914801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:36.915332 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:36.915642 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:36.916048 (+ 406us) server_negotiation.cc:299] Negotiation successful
0621 02:39:36.916461 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":394,"threads_started":1}
I20260621 02:39:36.922380 10738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea8f36f875d44e3bc1b6ee12197e647" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40077 } }
I20260621 02:39:36.922722 10738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:36.922821 10738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea8f36f875d44e3bc1b6ee12197e647, State: Initialized, Role: FOLLOWER
I20260621 02:39:36.923310 10738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea8f36f875d44e3bc1b6ee12197e647" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40077 } }
I20260621 02:39:36.923583 10738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:39:36.923719 10738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:39:36.923892 10738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:36.926308 10738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea8f36f875d44e3bc1b6ee12197e647" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40077 } }
I20260621 02:39:36.926819 10738 leader_election.cc:302] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ea8f36f875d44e3bc1b6ee12197e647; no voters:
I20260621 02:39:36.927469 10738 leader_election.cc:288] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:39:36.927583 10743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:36.930505 10738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:39:36.930903 10743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [term 1 LEADER]: Becoming Leader. State: Replica: 0ea8f36f875d44e3bc1b6ee12197e647, State: Running, Role: LEADER
I20260621 02:39:36.931384 10743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea8f36f875d44e3bc1b6ee12197e647" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40077 } }
I20260621 02:39:36.934669 10744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ea8f36f875d44e3bc1b6ee12197e647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea8f36f875d44e3bc1b6ee12197e647" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40077 } } }
I20260621 02:39:36.934834 10745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ea8f36f875d44e3bc1b6ee12197e647. Latest consensus state: current_term: 1 leader_uuid: "0ea8f36f875d44e3bc1b6ee12197e647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea8f36f875d44e3bc1b6ee12197e647" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40077 } } }
I20260621 02:39:36.935170 10744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:36.935230 10745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647 [sys.catalog]: This master's current role is: LEADER
I20260621 02:39:36.938253 10754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:39:36.942879 10754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:39:36.951241 10754 catalog_manager.cc:1347] Generated new cluster ID: 8a29140617ce4576b811740cc28f8c93
I20260621 02:39:36.951390 10754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:39:36.972805 10754 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:39:36.974174 10754 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:39:36.983110 10754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647: Generated new TSK 0
I20260621 02:39:36.984052 10754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:39:37.050421 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40077
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:37.391050 10764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:37.391319 10764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:37.391410 10764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:37.398387 10764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:37.398490 10764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:37.398603 10764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:39:37.407791 10764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:37.408923 10764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:37.410260 10764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:37.423254 10769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:37.423229 10772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:37.423195 10770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:37.423957 10764 server_base.cc:1034] running on GCE node
I20260621 02:39:37.424539 10764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:37.425450 10764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:37.426666 10764 hybrid_clock.cc:648] HybridClock initialized: now 1782009577426612 us; error 44 us; skew 500 ppm
I20260621 02:39:37.426942 10764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:37 dist-test-slave-jkn7 krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:37.429962 10764 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:39:37.431404 10764 webserver.cc:460] Webserver started at http://127.0.97.1:40031/ using document root <none> and password file <none>
I20260621 02:39:37.432005 10764 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:37.432096 10764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:37.432310 10764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:37.435248 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "72172feac9a449c984c117c3556c1f10"
format_stamp: "Formatted at 2026-06-21 02:39:37 on dist-test-slave-jkn7"
server_key: "a4d09e5050de986d44fb672b14e6ff52"
server_key_iv: "80754de705812b08d0cfd66b7d8fb21a"
server_key_version: "encryptionkey@0"
I20260621 02:39:37.435901 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "72172feac9a449c984c117c3556c1f10"
format_stamp: "Formatted at 2026-06-21 02:39:37 on dist-test-slave-jkn7"
server_key: "a4d09e5050de986d44fb672b14e6ff52"
server_key_iv: "80754de705812b08d0cfd66b7d8fb21a"
server_key_version: "encryptionkey@0"
I20260621 02:39:37.440675 10764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:39:37.443913 10779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:37.445178 10764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:39:37.445358 10764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "72172feac9a449c984c117c3556c1f10"
format_stamp: "Formatted at 2026-06-21 02:39:37 on dist-test-slave-jkn7"
server_key: "a4d09e5050de986d44fb672b14e6ff52"
server_key_iv: "80754de705812b08d0cfd66b7d8fb21a"
server_key_version: "encryptionkey@0"
I20260621 02:39:37.445545 10764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:37.457465 10764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:37.461092 10764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:37.461422 10764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:37.462293 10764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:37.463480 10764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:37.463587 10764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:37.463732 10764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:37.463804 10764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:37.496011 10764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36769
I20260621 02:39:37.496094 10891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36769 every 8 connection(s)
I20260621 02:39:37.497218 10764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:39:37.503230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 10764
I20260621 02:39:37.503464 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:39:37.504060 388 external_mini_cluster.cc:1428] Setting key 8efab47a7af4b2476ed14d013eccd578
Jun 21 02:39:37 dist-test-slave-jkn7 krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:39:37.518306 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:37.499678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36467 (local address 127.0.97.62:40077)
0621 02:39:37.500070 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:37.500074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:37.501044 (+ 970us) server_negotiation.cc:407] Connection header received
0621 02:39:37.502852 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:37.502858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:37.502953 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:37.503072 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:37.504666 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:37.505457 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:37.507042 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:37.507428 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:37.507563 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:37.511204 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:37.511209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:37.511212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:37.511245 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:37.514798 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:37.515616 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:37.515624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:37.515631 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:37.515788 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:37.516216 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:37.516223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:37.516229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:37.516607 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:37.516754 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:37.517559 (+ 805us) server_negotiation.cc:299] Negotiation successful
0621 02:39:37.517810 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":194,"threads_started":1}
I20260621 02:39:37.518970 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:37.500189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40077 (local address 127.0.97.1:36467)
0621 02:39:37.500860 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:37.500919 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:37.502592 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:37.503318 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:37.503330 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:37.503904 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:37.504501 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:37.504518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:37.505631 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:37.505635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:37.506863 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:37.506875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:37.507652 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:37.507659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:37.507901 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:37.508646 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:37.508671 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:37.510961 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:37.514987 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:37.514995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:37.515010 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:37.515480 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:37.515914 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:37.515916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:37.515919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:37.516100 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:37.516748 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:37.516754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:37.517082 (+ 328us) client_negotiation.cc:769] Sending connection context
0621 02:39:37.517425 (+ 343us) client_negotiation.cc:240] Negotiation successful
0621 02:39:37.517829 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":264,"threads_started":1}
I20260621 02:39:37.520517 10892 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40077
I20260621 02:39:37.520836 10892 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:37.521664 10892 heartbeater.cc:511] Master 127.0.97.62:40077 requested a full tablet report, sending...
I20260621 02:39:37.523783 10703 ts_manager.cc:194] Registered new tserver with Master: 72172feac9a449c984c117c3556c1f10 (127.0.97.1:36769)
I20260621 02:39:37.525193 10703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36467
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:39:37.552323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40077
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:37.863718 10898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:37.864054 10898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:37.864224 10898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:37.871400 10898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:37.871585 10898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:37.871771 10898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:39:37.881505 10898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:37.882867 10898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:37.884157 10898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:37.897722 10904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:37.897854 10906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:37.898351 10903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:37.906231 10898 server_base.cc:1034] running on GCE node
I20260621 02:39:37.906931 10898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:37.908006 10898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:37.909250 10898 hybrid_clock.cc:648] HybridClock initialized: now 1782009577909201 us; error 45 us; skew 500 ppm
I20260621 02:39:37.909632 10898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:37 dist-test-slave-jkn7 krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:37.915779 10898 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:39:37.917506 10898 webserver.cc:460] Webserver started at http://127.0.97.2:42137/ using document root <none> and password file <none>
I20260621 02:39:37.918272 10898 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:37.918428 10898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:37.918735 10898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:37.921725 10898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "a973c5c24e8e4bff9472c389afdf5b7c"
format_stamp: "Formatted at 2026-06-21 02:39:37 on dist-test-slave-jkn7"
server_key: "d434ec088c158be0fb2eb58ff6bd167e"
server_key_iv: "6a01e2a2295d20a32ec7a0022f6a7dcd"
server_key_version: "encryptionkey@0"
I20260621 02:39:37.922510 10898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a973c5c24e8e4bff9472c389afdf5b7c"
format_stamp: "Formatted at 2026-06-21 02:39:37 on dist-test-slave-jkn7"
server_key: "d434ec088c158be0fb2eb58ff6bd167e"
server_key_iv: "6a01e2a2295d20a32ec7a0022f6a7dcd"
server_key_version: "encryptionkey@0"
I20260621 02:39:37.927472 10898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:39:37.931093 10913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:37.932900 10898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:39:37.933071 10898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "a973c5c24e8e4bff9472c389afdf5b7c"
format_stamp: "Formatted at 2026-06-21 02:39:37 on dist-test-slave-jkn7"
server_key: "d434ec088c158be0fb2eb58ff6bd167e"
server_key_iv: "6a01e2a2295d20a32ec7a0022f6a7dcd"
server_key_version: "encryptionkey@0"
I20260621 02:39:37.933312 10898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:37.962692 10898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:37.966498 10898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:37.966732 10898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:37.967592 10898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:37.969434 10898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:37.969503 10898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:37.969584 10898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:37.969633 10898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:38.003129 10898 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35083
I20260621 02:39:38.003228 11025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35083 every 8 connection(s)
I20260621 02:39:38.004402 10898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:39:38.012214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 10898
I20260621 02:39:38.012431 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:39:38.012964 388 external_mini_cluster.cc:1428] Setting key fe1ec622a63fa1cad1049fa5dc973c54
Jun 21 02:39:38 dist-test-slave-jkn7 krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:39:38.025233 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.006895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49395 (local address 127.0.97.62:40077)
0621 02:39:38.007097 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.007103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.008299 (+ 1196us) server_negotiation.cc:407] Connection header received
0621 02:39:38.010071 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.010075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.010138 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.010223 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:38.011621 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.012310 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.014350 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.014893 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.015015 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.018472 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:38.018480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:38.018487 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:38.018530 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:38.022249 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:38.023007 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:38.023013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:38.023017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:38.023142 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:38.023568 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:38.023574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:38.023577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:38.023835 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:38.023985 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.024672 (+ 687us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.024887 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:39:38.026232 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.007435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40077 (local address 127.0.97.2:49395)
0621 02:39:38.008133 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:38.008176 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:38.009834 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:38.010410 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:38.010418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:38.010894 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:38.011433 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.011449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.012516 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.012521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.014212 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.014225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.015103 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.015108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.015297 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.015963 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:38.015984 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:38.018232 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:38.022429 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:38.022439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:38.022456 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:38.022880 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:38.023255 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:38.023261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:38.023265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:38.023452 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:38.023960 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:38.023969 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:38.024296 (+ 327us) client_negotiation.cc:769] Sending connection context
0621 02:39:38.024560 (+ 264us) client_negotiation.cc:240] Negotiation successful
0621 02:39:38.025088 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":303,"threads_started":1}
I20260621 02:39:38.027768 11026 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40077
I20260621 02:39:38.028098 11026 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:38.028935 11026 heartbeater.cc:511] Master 127.0.97.62:40077 requested a full tablet report, sending...
I20260621 02:39:38.030678 10703 ts_manager.cc:194] Registered new tserver with Master: a973c5c24e8e4bff9472c389afdf5b7c (127.0.97.2:35083)
I20260621 02:39:38.031937 10703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49395
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:39:38.063709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40077
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:39:38.384745 11031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:39:38.385010 11031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:39:38.385111 11031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:39:38.392194 11031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:39:38.392316 11031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:39:38.392428 11031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:39:38.401621 11031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:39:38.402842 11031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:39:38.404057 11031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:39:38.416721 11036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:38.416673 11037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:39:38.416751 11039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:39:38.417857 11031 server_base.cc:1034] running on GCE node
I20260621 02:39:38.418413 11031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:39:38.419263 11031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:39:38.420502 11031 hybrid_clock.cc:648] HybridClock initialized: now 1782009578420438 us; error 51 us; skew 500 ppm
I20260621 02:39:38.420821 11031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:39:38 dist-test-slave-jkn7 krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:39:38.424008 11031 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:39:38.425257 11031 webserver.cc:460] Webserver started at http://127.0.97.3:44165/ using document root <none> and password file <none>
I20260621 02:39:38.425917 11031 fs_manager.cc:359] Metadata directory not provided
I20260621 02:39:38.426012 11031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:39:38.426249 11031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:39:38.428857 11031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "6dad8a40293c42379ec87c6ef9a383f6"
format_stamp: "Formatted at 2026-06-21 02:39:38 on dist-test-slave-jkn7"
server_key: "0aebd97138fb8115b63c544d9d0d1080"
server_key_iv: "9f0f18c00013e7cc5ff6c88fea51f1a6"
server_key_version: "encryptionkey@0"
I20260621 02:39:38.429605 11031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6dad8a40293c42379ec87c6ef9a383f6"
format_stamp: "Formatted at 2026-06-21 02:39:38 on dist-test-slave-jkn7"
server_key: "0aebd97138fb8115b63c544d9d0d1080"
server_key_iv: "9f0f18c00013e7cc5ff6c88fea51f1a6"
server_key_version: "encryptionkey@0"
I20260621 02:39:38.434442 11031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:39:38.437739 11046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:38.438984 11031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:39:38.439113 11031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "6dad8a40293c42379ec87c6ef9a383f6"
format_stamp: "Formatted at 2026-06-21 02:39:38 on dist-test-slave-jkn7"
server_key: "0aebd97138fb8115b63c544d9d0d1080"
server_key_iv: "9f0f18c00013e7cc5ff6c88fea51f1a6"
server_key_version: "encryptionkey@0"
I20260621 02:39:38.439263 11031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:39:38.459391 11031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:39:38.463052 11031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:39:38.463312 11031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:39:38.464183 11031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:39:38.465371 11031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:39:38.465440 11031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:38.465523 11031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:39:38.465566 11031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:39:38.495926 11031 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37993
I20260621 02:39:38.495968 11158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37993 every 8 connection(s)
I20260621 02:39:38.497388 11031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:39:38.504124 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 11031
I20260621 02:39:38.504323 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:39:38.504747 388 external_mini_cluster.cc:1428] Setting key 20c1f35b12d1ab3f9c167e67b7273aaa
Jun 21 02:39:38 dist-test-slave-jkn7 krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:39:38.517434 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.500152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50697 (local address 127.0.97.62:40077)
0621 02:39:38.500363 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.500369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.501799 (+ 1430us) server_negotiation.cc:407] Connection header received
0621 02:39:38.503568 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.503574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.503666 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.503781 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:38.505548 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.506583 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.508289 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.508676 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.508771 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.512212 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:38.512216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:38.512220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:38.512252 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:38.514692 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:38.515383 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:38.515386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:38.515388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:38.515507 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:38.515901 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:38.515904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:38.515906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:38.516157 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:38.516262 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.516942 (+ 680us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.517111 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:39:38.518697 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.500527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40077 (local address 127.0.97.3:50697)
0621 02:39:38.501635 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:38.501680 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:38.503303 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:38.504026 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:38.504033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:38.504649 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:39:38.505286 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.505342 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.506726 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.506730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.508148 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.508159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.508854 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.508858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.509061 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.509795 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:39:38.509822 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:39:38.511993 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:39:38.514832 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:38.514840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:38.514855 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:38.515270 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:38.515611 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:39:38.515615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:39:38.515617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:39:38.515800 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:39:38.516260 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:39:38.516265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:39:38.516593 (+ 328us) client_negotiation.cc:769] Sending connection context
0621 02:39:38.516822 (+ 229us) client_negotiation.cc:240] Negotiation successful
0621 02:39:38.517218 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":622,"threads_started":1}
I20260621 02:39:38.520146 11159 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40077
I20260621 02:39:38.520450 11159 heartbeater.cc:464] Registering TS with master...
I20260621 02:39:38.521204 11159 heartbeater.cc:511] Master 127.0.97.62:40077 requested a full tablet report, sending...
I20260621 02:39:38.522791 10702 ts_manager.cc:194] Registered new tserver with Master: 6dad8a40293c42379ec87c6ef9a383f6 (127.0.97.3:37993)
I20260621 02:39:38.523662 10702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50697
I20260621 02:39:38.528043 10892 heartbeater.cc:503] Master 127.0.97.62:40077 was elected leader, sending a full tablet report...
I20260621 02:39:38.532009 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:39:38.548350 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.536243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.0.97.62:40077)
0621 02:39:38.536404 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.536408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.536910 (+ 502us) server_negotiation.cc:407] Connection header received
0621 02:39:38.537119 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.537127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.537181 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.537277 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:38.538653 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.539341 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.540544 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.540943 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.541099 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.543406 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:38.543411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:38.543414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:38.543442 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:38.545822 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:38.546460 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:38.546467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:38.546471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:38.546593 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:38.546944 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:38.546950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:38.546954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:38.547248 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:38.547501 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.547837 (+ 336us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.548057 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:39:38.552222 10702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:39:38.558812 10702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:39:38.594128 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.570495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35083 (local address 127.0.0.1:46104)
0621 02:39:38.584914 (+ 14419us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:38.584944 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:38.585137 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:38.586528 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:38.586548 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:38.586596 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:38.587279 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.587292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.589146 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.589151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.591097 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.591111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.593019 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.593026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.593216 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.593267 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 02:39:38.593412 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 02:39:38.593621 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14255,"mutex_wait_us":782,"spinlock_wait_cycles":768,"thread_start_us":13893,"threads_started":1}
I20260621 02:39:38.594636 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.570889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.0.97.2:35083)
0621 02:39:38.585693 (+ 14804us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.585701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.585727 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:39:38.585801 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.585807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.586050 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.586248 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:38.587478 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.588960 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.591257 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.592520 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.592674 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.592881 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.594109 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.594296 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14611,"thread_start_us":15650,"threads_started":1}
I20260621 02:39:38.594870 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.570015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36769 (local address 127.0.0.1:57544)
0621 02:39:38.570753 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:38.570802 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:38.570961 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:38.587718 (+ 16757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:38.587725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:38.587753 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:38.588084 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.588095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.589967 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.589973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.592067 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.592076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.593947 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.593955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.594195 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.594249 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 02:39:38.594345 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 02:39:38.594472 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":29,"thread_start_us":330,"threads_started":1}
I20260621 02:39:38.595307 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.571152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37993 (local address 127.0.0.1:51694)
0621 02:39:38.585318 (+ 14166us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:38.585340 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:38.585542 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:38.586557 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:38.586560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:38.586603 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:38.587083 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.587101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.589015 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.589019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.591699 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.591713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.593786 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.593793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.594612 (+ 819us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.594652 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 02:39:38.594751 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 02:39:38.594920 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13986,"mutex_wait_us":725,"spinlock_wait_cycles":13568,"thread_start_us":13474,"threads_started":1}
I20260621 02:39:38.595578 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.570335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.0.97.1:36769)
0621 02:39:38.587103 (+ 16768us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.587111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.587139 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:39:38.587234 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.587243 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.587484 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.587702 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:38.588291 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.589820 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.592210 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.593712 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.593857 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.594105 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.595125 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.595253 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16588,"thread_start_us":16421,"threads_started":1}
I20260621 02:39:38.596037 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.571651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.0.97.3:37993)
0621 02:39:38.584167 (+ 12516us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.584182 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.585440 (+ 1258us) server_negotiation.cc:407] Connection header received
0621 02:39:38.585743 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.585754 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.586058 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.586244 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:38.587304 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.588841 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.591868 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.593267 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.593427 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.593640 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.595529 (+ 1889us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.595705 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12243,"thread_start_us":12546,"threads_started":1}
I20260621 02:39:38.599030 10827 tablet_service.cc:1467] Processing CreateTablet for tablet 187eb6b676c941318b3cb06292634765 (DEFAULT_TABLE table=test-table [id=8ba5634ec3c847d0a852c096c1573bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:38.599102 10961 tablet_service.cc:1467] Processing CreateTablet for tablet 187eb6b676c941318b3cb06292634765 (DEFAULT_TABLE table=test-table [id=8ba5634ec3c847d0a852c096c1573bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:38.599301 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 187eb6b676c941318b3cb06292634765 (DEFAULT_TABLE table=test-table [id=8ba5634ec3c847d0a852c096c1573bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:39:38.600376 10827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187eb6b676c941318b3cb06292634765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:38.600452 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187eb6b676c941318b3cb06292634765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:38.600468 10961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187eb6b676c941318b3cb06292634765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:39:38.612150 11177 tablet_bootstrap.cc:492] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: Bootstrap starting.
I20260621 02:39:38.615181 11178 tablet_bootstrap.cc:492] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: Bootstrap starting.
I20260621 02:39:38.616799 11179 tablet_bootstrap.cc:492] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: Bootstrap starting.
I20260621 02:39:38.617237 11177 tablet_bootstrap.cc:654] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:38.618660 11177 log.cc:824] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:38.619164 11178 tablet_bootstrap.cc:654] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:38.619794 11179 tablet_bootstrap.cc:654] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: Neither blocks nor log segments found. Creating new log.
I20260621 02:39:38.620673 11178 log.cc:824] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:38.620728 11179 log.cc:824] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:39:38.624198 11179 tablet_bootstrap.cc:492] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: No bootstrap required, opened a new log
I20260621 02:39:38.624221 11178 tablet_bootstrap.cc:492] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: No bootstrap required, opened a new log
I20260621 02:39:38.624488 11178 ts_tablet_manager.cc:1397] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:39:38.624509 11179 ts_tablet_manager.cc:1397] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260621 02:39:38.624621 11177 tablet_bootstrap.cc:492] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: No bootstrap required, opened a new log
I20260621 02:39:38.624912 11177 ts_tablet_manager.cc:1397] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260621 02:39:38.632027 11178 raft_consensus.cc:348] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } }
I20260621 02:39:38.632412 11178 raft_consensus.cc:374] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:38.632510 11178 raft_consensus.cc:729] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad8a40293c42379ec87c6ef9a383f6, State: Initialized, Role: FOLLOWER
I20260621 02:39:38.633229 11178 consensus_queue.cc:260] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } }
I20260621 02:39:38.634496 11159 heartbeater.cc:503] Master 127.0.97.62:40077 was elected leader, sending a full tablet report...
I20260621 02:39:38.635124 11178 ts_tablet_manager.cc:1428] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260621 02:39:38.635492 11177 raft_consensus.cc:348] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } }
I20260621 02:39:38.635492 11179 raft_consensus.cc:348] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } }
I20260621 02:39:38.635882 11177 raft_consensus.cc:374] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:38.635989 11179 raft_consensus.cc:374] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:39:38.635978 11177 raft_consensus.cc:729] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72172feac9a449c984c117c3556c1f10, State: Initialized, Role: FOLLOWER
I20260621 02:39:38.636083 11179 raft_consensus.cc:729] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a973c5c24e8e4bff9472c389afdf5b7c, State: Initialized, Role: FOLLOWER
I20260621 02:39:38.636608 11177 consensus_queue.cc:260] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } }
I20260621 02:39:38.636700 11179 consensus_queue.cc:260] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } }
I20260621 02:39:38.638233 11026 heartbeater.cc:503] Master 127.0.97.62:40077 was elected leader, sending a full tablet report...
I20260621 02:39:38.638531 11177 ts_tablet_manager.cc:1428] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260621 02:39:38.638793 11179 ts_tablet_manager.cc:1428] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
W20260621 02:39:38.748978 11160 tablet.cc:2369] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:39:38.749524 10893 tablet.cc:2369] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:39:38.756315 11027 tablet.cc:2369] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:39:38.850723 11184 raft_consensus.cc:482] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:39:38.851008 11184 raft_consensus.cc:504] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } }
I20260621 02:39:38.852640 11184 leader_election.cc:288] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dad8a40293c42379ec87c6ef9a383f6 (127.0.97.3:37993), a973c5c24e8e4bff9472c389afdf5b7c (127.0.97.2:35083)
I20260621 02:39:38.860317 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.852881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35083 (local address 127.0.97.1:48299)
0621 02:39:38.853778 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:38.853796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:38.854002 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:38.854612 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:38.854619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:38.854648 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:38.854950 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.854963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.856667 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.856673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.858310 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.858321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.859507 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.859512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.859692 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.859723 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:39:38.859820 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 02:39:38.859941 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":64,"thread_start_us":563,"threads_started":1}
I20260621 02:39:38.860373 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.853177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48299 (local address 127.0.97.2:35083)
0621 02:39:38.853781 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.853788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.853869 (+ 81us) server_negotiation.cc:407] Connection header received
0621 02:39:38.854222 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.854228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.854312 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.854380 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:38.855127 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.856516 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.858454 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.859291 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.859446 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.859544 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.859871 (+ 327us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.860043 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475}
I20260621 02:39:38.861606 10981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187eb6b676c941318b3cb06292634765" candidate_uuid: "72172feac9a449c984c117c3556c1f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" is_pre_election: true
I20260621 02:39:38.861945 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.853090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37993 (local address 127.0.97.1:43639)
0621 02:39:38.854409 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0621 02:39:38.854426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:39:38.854585 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:39:38.855002 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:39:38.855010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:39:38.855035 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:39:38.855412 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.855427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.856805 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.856813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.859471 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:39:38.859493 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.861129 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.861137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:39:38.861327 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.861360 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:39:38.861441 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 02:39:38.861564 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":28,"thread_start_us":870,"threads_started":1}
I20260621 02:39:38.862201 10981 raft_consensus.cc:2454] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72172feac9a449c984c117c3556c1f10 in term 0.
I20260621 02:39:38.862674 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.853638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43639 (local address 127.0.97.3:37993)
0621 02:39:38.853867 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.853876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.854492 (+ 616us) server_negotiation.cc:407] Connection header received
0621 02:39:38.854718 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.854725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.854822 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.854936 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:39:38.855602 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.856649 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.859717 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.860934 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.862131 (+ 1197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.862236 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.862287 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.862407 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:39:38.862905 10782 leader_election.cc:302] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72172feac9a449c984c117c3556c1f10, a973c5c24e8e4bff9472c389afdf5b7c; no voters:
I20260621 02:39:38.863319 11184 raft_consensus.cc:2781] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:39:38.863466 11184 raft_consensus.cc:482] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:39:38.863564 11184 raft_consensus.cc:3037] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:38.863590 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187eb6b676c941318b3cb06292634765" candidate_uuid: "72172feac9a449c984c117c3556c1f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad8a40293c42379ec87c6ef9a383f6" is_pre_election: true
I20260621 02:39:38.864068 11114 raft_consensus.cc:2454] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72172feac9a449c984c117c3556c1f10 in term 0.
I20260621 02:39:38.865962 11184 raft_consensus.cc:504] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } }
I20260621 02:39:38.866510 11184 leader_election.cc:288] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [CANDIDATE]: Term 1 election: Requested vote from peers 6dad8a40293c42379ec87c6ef9a383f6 (127.0.97.3:37993), a973c5c24e8e4bff9472c389afdf5b7c (127.0.97.2:35083)
I20260621 02:39:38.866925 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187eb6b676c941318b3cb06292634765" candidate_uuid: "72172feac9a449c984c117c3556c1f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad8a40293c42379ec87c6ef9a383f6"
I20260621 02:39:38.867089 11114 raft_consensus.cc:3037] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:38.869046 11114 raft_consensus.cc:2454] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72172feac9a449c984c117c3556c1f10 in term 1.
I20260621 02:39:38.869387 10981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187eb6b676c941318b3cb06292634765" candidate_uuid: "72172feac9a449c984c117c3556c1f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a973c5c24e8e4bff9472c389afdf5b7c"
I20260621 02:39:38.869431 10780 leader_election.cc:302] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad8a40293c42379ec87c6ef9a383f6, 72172feac9a449c984c117c3556c1f10; no voters:
I20260621 02:39:38.869597 10981 raft_consensus.cc:3037] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:39:38.869688 11184 raft_consensus.cc:2781] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:39:38.870157 11184 raft_consensus.cc:686] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 1 LEADER]: Becoming Leader. State: Replica: 72172feac9a449c984c117c3556c1f10, State: Running, Role: LEADER
I20260621 02:39:38.870652 11184 consensus_queue.cc:237] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } }
I20260621 02:39:38.872859 10981 raft_consensus.cc:2454] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72172feac9a449c984c117c3556c1f10 in term 1.
I20260621 02:39:38.875936 10703 catalog_manager.cc:5495] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72172feac9a449c984c117c3556c1f10 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "72172feac9a449c984c117c3556c1f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72172feac9a449c984c117c3556c1f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 } health_report { overall_health: UNKNOWN } } }
I20260621 02:39:38.894786 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.888770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.0.97.1:36769)
0621 02:39:38.889078 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.889085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.889102 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:39:38.889185 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.889191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.889329 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.889458 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:39:38.889852 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.890495 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.892300 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.892688 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.892776 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.893446 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:39:38.894113 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:39:38.894263 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.894364 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.894456 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260621 02:39:38.904507 11114 raft_consensus.cc:1261] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [term 1 FOLLOWER]: Refusing update from remote peer 72172feac9a449c984c117c3556c1f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:38.904520 10981 raft_consensus.cc:1261] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [term 1 FOLLOWER]: Refusing update from remote peer 72172feac9a449c984c117c3556c1f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:39:38.905509 11184 consensus_queue.cc:1035] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad8a40293c42379ec87c6ef9a383f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:38.906176 11184 consensus_queue.cc:1035] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a973c5c24e8e4bff9472c389afdf5b7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:39:38.918076 11191 mvcc.cc:205] Tried to move back new op lower bound from 7299111235183112192 to 7299111235058061312. Current Snapshot: MvccSnapshot[applied={T|T < 7299111235183112192}]
I20260621 02:39:38.918547 11192 mvcc.cc:205] Tried to move back new op lower bound from 7299111235183112192 to 7299111235058061312. Current Snapshot: MvccSnapshot[applied={T|T < 7299111235183112192}]
I20260621 02:39:38.945696 10703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:38.945960 10703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:39:38.949090 10703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ea8f36f875d44e3bc1b6ee12197e647: Sending DeleteTablet for 3 replicas of tablet 187eb6b676c941318b3cb06292634765
I20260621 02:39:38.950112 10827 tablet_service.cc:1514] Processing DeleteTablet for tablet 187eb6b676c941318b3cb06292634765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57544
I20260621 02:39:38.950433 10961 tablet_service.cc:1514] Processing DeleteTablet for tablet 187eb6b676c941318b3cb06292634765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46104
I20260621 02:39:38.950713 11094 tablet_service.cc:1514] Processing DeleteTablet for tablet 187eb6b676c941318b3cb06292634765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:39:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51694
I20260621 02:39:38.951927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 10764
I20260621 02:39:38.957688 11203 tablet_replica.cc:330] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: stopping tablet replica
I20260621 02:39:38.958151 11203 raft_consensus.cc:2229] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:38.958686 11203 raft_consensus.cc:2258] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:38.959432 11201 tablet_replica.cc:330] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: stopping tablet replica
I20260621 02:39:38.959741 11201 raft_consensus.cc:2229] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:39:38.960196 11201 raft_consensus.cc:2258] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:38.962353 11203 ts_tablet_manager.cc:1905] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:38.966432 11201 ts_tablet_manager.cc:1905] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:39:38.967024 11202 tablet_replica.cc:330] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: stopping tablet replica
I20260621 02:39:38.967496 11203 ts_tablet_manager.cc:1918] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:38.967644 11203 log.cc:1196] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/187eb6b676c941318b3cb06292634765
I20260621 02:39:38.968125 11203 ts_tablet_manager.cc:1939] T 187eb6b676c941318b3cb06292634765 P 6dad8a40293c42379ec87c6ef9a383f6: Deleting consensus metadata
I20260621 02:39:38.968076 11202 raft_consensus.cc:2229] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:39:38.968514 11202 raft_consensus.cc:2258] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:39:38.969753 10687 catalog_manager.cc:4841] TS 6dad8a40293c42379ec87c6ef9a383f6 (127.0.97.3:37993): tablet 187eb6b676c941318b3cb06292634765 (table test-table [id=8ba5634ec3c847d0a852c096c1573bbb]) successfully deleted
I20260621 02:39:38.970539 11202 ts_tablet_manager.cc:1905] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 02:39:38 dist-test-slave-jkn7 krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:39:38.976539 11202 ts_tablet_manager.cc:1918] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:38.976701 11202 log.cc:1196] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/187eb6b676c941318b3cb06292634765
I20260621 02:39:38.977195 11202 ts_tablet_manager.cc:1939] T 187eb6b676c941318b3cb06292634765 P a973c5c24e8e4bff9472c389afdf5b7c: Deleting consensus metadata
I20260621 02:39:38.978876 10689 catalog_manager.cc:4841] TS a973c5c24e8e4bff9472c389afdf5b7c (127.0.97.2:35083): tablet 187eb6b676c941318b3cb06292634765 (table test-table [id=8ba5634ec3c847d0a852c096c1573bbb]) successfully deleted
I20260621 02:39:38.980609 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:38.953611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.97.1:36769)
0621 02:39:38.953824 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:38.953838 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:38.960840 (+ 7002us) server_negotiation.cc:407] Connection header received
0621 02:39:38.960954 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:38.960962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:38.961061 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:38.961189 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:38.963341 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.964752 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.967093 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:38.967532 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:38.967672 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:38.972517 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:38.972528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:38.972544 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:38.972591 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:38.976797 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:38.977780 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:38.977787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:38.977791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:38.977956 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:38.978439 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:38.978444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:38.978447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:38.979325 (+ 878us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:38.979618 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:38.980183 (+ 565us) server_negotiation.cc:299] Negotiation successful
0621 02:39:38.980361 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:39:38.981874 10887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:38.981918 11201 ts_tablet_manager.cc:1918] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:39:38.982124 11201 log.cc:1196] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/187eb6b676c941318b3cb06292634765
I20260621 02:39:39.076572 11201 ts_tablet_manager.cc:1939] T 187eb6b676c941318b3cb06292634765 P 72172feac9a449c984c117c3556c1f10: Deleting consensus metadata
I20260621 02:39:39.078114 10687 catalog_manager.cc:4841] TS 72172feac9a449c984c117c3556c1f10 (127.0.97.1:36769): tablet 187eb6b676c941318b3cb06292634765 (table test-table [id=8ba5634ec3c847d0a852c096c1573bbb]) successfully deleted
I20260621 02:39:39.081542 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 10764
I20260621 02:39:39.099253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 10898
Jun 21 02:39:39 dist-test-slave-jkn7 krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:39:39.113917 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:39.099952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.0.97.2:35083)
0621 02:39:39.100140 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:39.100149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:39.100226 (+ 77us) server_negotiation.cc:407] Connection header received
0621 02:39:39.100425 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:39.100435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:39.100512 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:39.100616 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:39.101848 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:39.102559 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:39.103755 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:39.104151 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:39.104284 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:39.108237 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:39.108248 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:39.108262 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:39.108295 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:39.111149 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:39.111780 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:39.111793 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:39.111795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:39.111911 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:39.112255 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:39.112261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:39.112264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:39.112622 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:39.112905 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:39.113203 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 02:39:39.113633 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:39:39.114538 11021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:39.203339 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 10898
I20260621 02:39:39.220484 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 11031
Jun 21 02:39:39 dist-test-slave-jkn7 krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:39:39.235220 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:39:39.221222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.0.97.3:37993)
0621 02:39:39.221416 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:39:39.221421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:39:39.221439 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:39:39.221558 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:39:39.221561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:39:39.221635 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:39:39.221720 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:39:39.223078 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:39.223783 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:39.225002 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:39:39.225480 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:39:39.225580 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:39:39.229849 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:39:39.229873 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:39:39.229890 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:39:39.229922 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:39:39.232670 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:39.233349 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:39.233358 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:39.233363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:39.233506 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:39:39.233854 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:39:39.233858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:39:39.233860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:39:39.234228 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:39:39.234442 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 02:39:39.234665 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 02:39:39.234859 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 02:39:39.235807 11154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:39.322422 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 11031
I20260621 02:39:39.341437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 10671
I20260621 02:39:39.342563 10733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:39:39.450109 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 10671
2026-06-21T02:39:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4268 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:39:39 dist-test-slave-jkn7 krb5kdc[11211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:39:39 dist-test-slave-jkn7 krb5kdc[11211](info): set up 2 sockets
Jun 21 02:39:39 dist-test-slave-jkn7 krb5kdc[11211](info): commencing operation
krb5kdc: starting...
W20260621 02:39:40.538094 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:39:40 dist-test-slave-jkn7 krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:39:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260621 02:39:40.839859 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskXxBWLs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/postgres -l logfile start
2026-06-21 02:39:43.114 UTC [11241] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 02:39:43.114 UTC [11241] LOG: listening on IPv4 address "127.0.97.20", port 43789
2026-06-21 02:39:43.117 UTC [11241] LOG: listening on Unix socket "/tmp/.s.PGSQL.43789"
2026-06-21 02:39:43.138 UTC [11244] LOG: database system was shut down at 2026-06-21 02:39:41 UTC
2026-06-21 02:39:43.142 UTC [11241] LOG: database system is ready to accept connections
I20260621 02:39:44.121927 388 mini_postgres.cc:96] Postgres bound to 43789
2026-06-21 02:39:44.138 UTC [11252] FATAL: database "slave" does not exist
127.0.97.20:43789 - accepting connections
I20260621 02:39:44.139037 388 mini_ranger.cc:161] Starting Ranger...
I20260621 02:39:44.162828 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 02:39:44.591413 388 mini_ranger.cc:87] Created ranger Postgres database
I20260621 02:39:44.591552 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-admin
2026-06-21 02:39:44,867 [I] DB FLAVOR :POSTGRES
2026-06-21 02:39:44,868 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:39:44,868 [I] Checking connection..
2026-06-21 02:39:44,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 02:39:45,129 [I] Checking connection passed.
2026-06-21 02:39:45,129 [I] --------- Verifying version history table ---------
2026-06-21 02:39:45,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:39:45,372 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:39:45,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:39:45,629 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:39:45,629 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 02:39:45,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 02:39:45.879 UTC [11316] WARNING: there is no transaction in progress
2026-06-21 02:39:45,885 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 02:39:45,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:39:46,125 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 02:39:46,125 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:39:46,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 02:39:46,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 02:39:46,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jkn7.c.gcp-upstream.internal','N') ;"
2026-06-21 02:39:46,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:39:47,111 [I] Table x_portal_user does not exist in database ranger
2026-06-21 02:39:47,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:39:47,338 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 02:39:47,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:39:47,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:39:47,815 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 02:39:47,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 02:39:48.734 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.750 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.766 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.782 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.797 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.813 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.867 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.874 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.888 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.898 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.911 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.924 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.933 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.939 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.945 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:48.952 UTC [11424] WARNING: there is no transaction in progress
2026-06-21 02:39:49,556 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 02:39:49,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 02:39:49,777 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 02:39:49,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:39:50,035 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 02:39:50,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:39:50,290 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 02:39:50,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:39:50,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:39:50,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jkn7.c.gcp-upstream.internal';"
2026-06-21 02:39:51,002 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 02:39:51,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 02:39:51,231 [I] DB_PATCHES have already been applied
I20260621 02:39:51.237432 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskXxBWLs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 02:39:51.237563 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260621 02:39:51.252547 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:39447
Jun 21, 2026 2:39:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:39:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:39:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 2:39:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 2:39:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 2:39:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 2:39:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 02:39:52.370862 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.232s user 0.001s sys 0.038s
Jun 21, 2026 2:40:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 2:40:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 2:40:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 2:40:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 2:40:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 02:40:13.914175 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 02:40:14 dist-test-slave-jkn7 krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009614, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 02:40:14.072005 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 02:40:14.091845 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 02:40:14.900328 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 02:40:14.900513 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-kms
I20260621 02:40:14.900637 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:43789
2026-06-21 02:40:15,329 [I] DB FLAVOR :POSTGRES
2026-06-21 02:40:15,329 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:40:15,329 [I] Checking connection
2026-06-21 02:40:15,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 02:40:15,580 [I] connection success
2026-06-21 02:40:15,580 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 02:40:15,580 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 02:40:15,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 02:40:15,839 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 02:40:15,839 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:40:15,839 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 02:40:15,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43789/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 02:40:16,082 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 02:40:16.533571 388 mini_ranger_kms.cc:310] Created kms service
I20260621 02:40:16.743875 388 mini_ranger_kms.cc:326] Created kudu user
I20260621 02:40:16.786846 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 02:40:16 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009614, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:40:17.194852 388 mini_ranger_kms.cc:384] Added ranger policy
I20260621 02:40:17.195255 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 02:40:17.195334 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260621 02:40:17.212280 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11604
I20260621 02:40:17.212482 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:37519
Jun 21, 2026 2:40:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:40:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:40:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 2:40:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 2:40:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 2:40:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 21 02:40:19 dist-test-slave-jkn7 krb5kdc[11211](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782009619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:40:19.473994 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.402s user 0.133s sys 0.312s
I20260621 02:40:19.474251 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 02:40:19.474478 388 mini_ranger_kms.cc:401] 127.0.97.20:37519/kms/v1/keys
Jun 21 02:40:19 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 21 02:40:20 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 21 02:40:22 dist-test-slave-jkn7 krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:40:22.232986 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42103
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42103
--ranger_config_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37519/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:40:22.554014 11655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:40:22.554278 11655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:40:22.554361 11655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:40:22.561136 11655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:40:22.561237 11655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:40:22.561329 11655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:40:22.561411 11655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:40:22.561460 11655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:40:22.571218 11655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:37519/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42103
--ranger_config_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:40:22.572371 11655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:40:22.574024 11655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:40:22.586014 11663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:40:22.586074 11660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:40:22.586035 11661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:40:22.586697 11655 server_base.cc:1034] running on GCE node
I20260621 02:40:22.587512 11655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:40:22.588860 11655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:40:22.590123 11655 hybrid_clock.cc:648] HybridClock initialized: now 1782009622590074 us; error 52 us; skew 500 ppm
I20260621 02:40:22.590485 11655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:40:22 dist-test-slave-jkn7 krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:40:22.597810 11655 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:40:22.599363 11655 webserver.cc:460] Webserver started at http://127.0.97.62:41947/ using document root <none> and password file <none>
I20260621 02:40:22.600042 11655 fs_manager.cc:359] Metadata directory not provided
I20260621 02:40:22.600179 11655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:40:22.600481 11655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:40:22 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:40:22.660034 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "e66522c9dec24afb81f31b1f7d61afc4"
format_stamp: "Formatted at 2026-06-21 02:40:22 on dist-test-slave-jkn7"
server_key: "054adc2b9d324d210169dae680d8fe43"
server_key_iv: "443f69a520fe9c8b8927aff9dbc4c1b7"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:22.661103 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "e66522c9dec24afb81f31b1f7d61afc4"
format_stamp: "Formatted at 2026-06-21 02:40:22 on dist-test-slave-jkn7"
server_key: "054adc2b9d324d210169dae680d8fe43"
server_key_iv: "443f69a520fe9c8b8927aff9dbc4c1b7"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:22.667990 11655 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 02:40:22.687729 11670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:40:22.689471 11655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:40:22.689661 11655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "e66522c9dec24afb81f31b1f7d61afc4"
format_stamp: "Formatted at 2026-06-21 02:40:22 on dist-test-slave-jkn7"
server_key: "054adc2b9d324d210169dae680d8fe43"
server_key_iv: "443f69a520fe9c8b8927aff9dbc4c1b7"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:22.689982 11655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:40:22.717541 11655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:40:22.722499 11655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:40:22.722815 11655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:40:22.743660 11721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42103 every 8 connection(s)
I20260621 02:40:22.743665 11655 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42103
I20260621 02:40:22.745381 11655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:40:22.747417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 11655
I20260621 02:40:22.747633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:40:22.748407 388 external_mini_cluster.cc:1428] Setting key 2f60f601b718670b2b43f0ccaaf2d469
I20260621 02:40:22.750018 11722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:40:22.768183 11722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4: Bootstrap starting.
I20260621 02:40:22.771859 11722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4: Neither blocks nor log segments found. Creating new log.
I20260621 02:40:22.773460 11722 log.cc:824] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4: Log is configured to *not* fsync() on all Append() calls
I20260621 02:40:22.776774 11722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4: No bootstrap required, opened a new log
Jun 21 02:40:22 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:40:22.785944 11722 raft_consensus.cc:348] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66522c9dec24afb81f31b1f7d61afc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42103 } }
I20260621 02:40:22.786451 11722 raft_consensus.cc:374] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:40:22.786566 11722 raft_consensus.cc:729] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e66522c9dec24afb81f31b1f7d61afc4, State: Initialized, Role: FOLLOWER
I20260621 02:40:22.787273 11722 consensus_queue.cc:260] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66522c9dec24afb81f31b1f7d61afc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42103 } }
I20260621 02:40:22.787544 11722 raft_consensus.cc:388] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:40:22.787704 11722 raft_consensus.cc:482] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:40:22.787884 11722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:40:22.790773 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:22.750411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.0.97.62:42103)
0621 02:40:22.765953 (+ 15542us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:22.765969 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:22.766030 (+ 61us) server_negotiation.cc:407] Connection header received
0621 02:40:22.768318 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:22.768347 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:22.768871 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:22.769301 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:40:22.770964 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:22.772967 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:22.774677 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:22.775707 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:22.775886 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:22.780677 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:40:22.780688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:40:22.780710 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:40:22.780749 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:40:22.784734 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:22.785579 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:22.785587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:22.785596 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:22.785768 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:22.786232 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:22.786237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:22.786240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:22.786930 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:40:22.787216 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:22.788533 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0621 02:40:22.788973 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15266,"thread_start_us":14994,"threads_started":1}
I20260621 02:40:22.791638 11722 raft_consensus.cc:504] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66522c9dec24afb81f31b1f7d61afc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42103 } }
I20260621 02:40:22.792192 11722 leader_election.cc:302] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e66522c9dec24afb81f31b1f7d61afc4; no voters:
I20260621 02:40:22.792917 11722 leader_election.cc:288] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:40:22.793071 11727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:40:22.798110 11727 raft_consensus.cc:686] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [term 1 LEADER]: Becoming Leader. State: Replica: e66522c9dec24afb81f31b1f7d61afc4, State: Running, Role: LEADER
I20260621 02:40:22.798753 11727 consensus_queue.cc:237] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66522c9dec24afb81f31b1f7d61afc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42103 } }
I20260621 02:40:22.799360 11722 sys_catalog.cc:564] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:40:22.801524 11728 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e66522c9dec24afb81f31b1f7d61afc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66522c9dec24afb81f31b1f7d61afc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42103 } } }
I20260621 02:40:22.801901 11728 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [sys.catalog]: This master's current role is: LEADER
I20260621 02:40:22.802443 11729 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e66522c9dec24afb81f31b1f7d61afc4. Latest consensus state: current_term: 1 leader_uuid: "e66522c9dec24afb81f31b1f7d61afc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66522c9dec24afb81f31b1f7d61afc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42103 } } }
I20260621 02:40:22.802708 11729 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4 [sys.catalog]: This master's current role is: LEADER
I20260621 02:40:22.806509 11734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:40:22.813102 11734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:40:22.824638 11734 catalog_manager.cc:1347] Generated new cluster ID: 05ff625288f14924a26c7fab50ccfaa4
I20260621 02:40:22.824753 11734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:40:22.838279 11722 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 02:40:22.864181 11734 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:40:22.865768 11734 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:40:22.875032 11734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4: Generated new TSK 0
I20260621 02:40:22.876128 11734 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:40:23.645356 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37519/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42103
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:40:23 dist-test-slave-jkn7 krb5kdc[11211](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782009623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 02:40:24.083101 11767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:40:24.083527 11767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:40:24.083706 11767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:40:24.094995 11767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:40:24.095214 11767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:40:24.095469 11767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:40:24.111675 11767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:37519/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:40:24.114045 11767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:40:24.115877 11767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:40:24.134322 11778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:40:24.134343 11777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:40:24.135676 11780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:40:24.135797 11767 server_base.cc:1034] running on GCE node
I20260621 02:40:24.136437 11767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:40:24.137554 11767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:40:24.138813 11767 hybrid_clock.cc:648] HybridClock initialized: now 1782009624138725 us; error 79 us; skew 500 ppm
I20260621 02:40:24.139118 11767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:40:24 dist-test-slave-jkn7 krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:40:24.142584 11767 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:40:24.144089 11767 webserver.cc:460] Webserver started at http://127.0.97.1:41619/ using document root <none> and password file <none>
I20260621 02:40:24.144738 11767 fs_manager.cc:359] Metadata directory not provided
I20260621 02:40:24.144833 11767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:40:24.145071 11767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:40:24 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:40:24.179932 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "3ac6b10cbe734b56a40865d6c28dd760"
format_stamp: "Formatted at 2026-06-21 02:40:24 on dist-test-slave-jkn7"
server_key: "d0dd16e35202f4dbf2dc6cc3db2f882f"
server_key_iv: "f0cc56cb5268bee3c9b2e08a8d6e4f77"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:24.183387 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3ac6b10cbe734b56a40865d6c28dd760"
format_stamp: "Formatted at 2026-06-21 02:40:24 on dist-test-slave-jkn7"
server_key: "d0dd16e35202f4dbf2dc6cc3db2f882f"
server_key_iv: "f0cc56cb5268bee3c9b2e08a8d6e4f77"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:24.197068 11767 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.011s
I20260621 02:40:24.238022 11787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:40:24.240221 11767 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.013s
I20260621 02:40:24.240542 11767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "3ac6b10cbe734b56a40865d6c28dd760"
format_stamp: "Formatted at 2026-06-21 02:40:24 on dist-test-slave-jkn7"
server_key: "d0dd16e35202f4dbf2dc6cc3db2f882f"
server_key_iv: "f0cc56cb5268bee3c9b2e08a8d6e4f77"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:24.240942 11767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:40:24.287530 11767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:40:24.293083 11767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:40:24.293517 11767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:40:24.301069 11767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:40:24.309355 11767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:40:24.309540 11767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260621 02:40:24.309772 11767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:40:24.309930 11767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:40:24 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:40:24.698200 11899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40913 every 8 connection(s)
I20260621 02:40:24.698613 11767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40913
I20260621 02:40:24.700469 11767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:40:24.701524 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 11767
I20260621 02:40:24.701716 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:40:24.702484 388 external_mini_cluster.cc:1428] Setting key faf73cc97828def1d8f646e9f105a205
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Jun 21 02:40:24 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260621 02:40:24.777565 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:24.720000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55665 (local address 127.0.97.62:42103)
0621 02:40:24.755305 (+ 35305us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:24.755312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:24.758783 (+ 3471us) server_negotiation.cc:407] Connection header received
0621 02:40:24.758852 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:24.758857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:24.758934 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:24.759030 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:40:24.761981 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:24.763122 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:24.765415 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:24.766020 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:24.766128 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:24.770465 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:40:24.770473 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:40:24.770477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:40:24.770529 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:40:24.773899 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:24.774828 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:24.774834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:24.774838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:24.774974 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:24.775513 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:24.775518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:24.775521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:24.775852 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:40:24.775974 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:24.776935 (+ 961us) server_negotiation.cc:299] Negotiation successful
0621 02:40:24.777175 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35162,"thread_start_us":35322,"threads_started":1}
I20260621 02:40:24.779817 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:24.719448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42103 (local address 127.0.97.1:55665)
0621 02:40:24.755938 (+ 36490us) negotiation.cc:107] Waiting for socket to connect
0621 02:40:24.755996 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 02:40:24.758330 (+ 2334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:40:24.760260 (+ 1930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:40:24.760272 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:40:24.760890 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:40:24.761776 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:24.761800 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:24.763328 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:24.763333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:24.765225 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:24.765240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:24.766454 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:24.766460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:24.766734 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:24.767541 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:40:24.767574 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:40:24.770190 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:40:24.774101 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:40:24.774111 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:40:24.774129 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:40:24.774691 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:40:24.775128 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:40:24.775132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:40:24.775136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:40:24.775389 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:40:24.776084 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:40:24.776091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:40:24.776507 (+ 416us) client_negotiation.cc:769] Sending connection context
0621 02:40:24.777859 (+ 1352us) client_negotiation.cc:240] Negotiation successful
0621 02:40:24.778442 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36168,"thread_start_us":39633,"threads_started":1}
I20260621 02:40:24.781689 11900 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42103
I20260621 02:40:24.782011 11900 heartbeater.cc:464] Registering TS with master...
I20260621 02:40:24.782891 11900 heartbeater.cc:511] Master 127.0.97.62:42103 requested a full tablet report, sending...
I20260621 02:40:24.785356 11687 ts_manager.cc:194] Registered new tserver with Master: 3ac6b10cbe734b56a40865d6c28dd760 (127.0.97.1:40913)
I20260621 02:40:24.787315 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260621 02:40:24.816190 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.111s user 0.004s sys 0.013s
I20260621 02:40:24.817106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37519/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42103
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:40:24 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260621 02:40:25.283257 11906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:40:25.283595 11906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:40:25.283766 11906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:40:25.291340 11906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:40:25.291482 11906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:40:25.291671 11906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:40:25.302000 11906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:37519/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:40:25.303455 11906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:40:25.304737 11906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:40:25.321532 11920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:40:25.322701 11921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:40:25.324105 11924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:40:25.324062 11906 server_base.cc:1034] running on GCE node
I20260621 02:40:25.324788 11906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:40:25.326025 11906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:40:25.327247 11906 hybrid_clock.cc:648] HybridClock initialized: now 1782009625327184 us; error 61 us; skew 500 ppm
I20260621 02:40:25.327587 11906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:40:25 dist-test-slave-jkn7 krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:40:25.330808 11906 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:40:25.332489 11906 webserver.cc:460] Webserver started at http://127.0.97.2:35393/ using document root <none> and password file <none>
I20260621 02:40:25.333199 11906 fs_manager.cc:359] Metadata directory not provided
I20260621 02:40:25.333356 11906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:40:25.333678 11906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:40:25 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:40:25.350533 11906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "98de37a5a10a45dc889965d52ed26fcd"
format_stamp: "Formatted at 2026-06-21 02:40:25 on dist-test-slave-jkn7"
server_key: "ebfaa73902ef8df8268ec74d68ed411c"
server_key_iv: "f05923a6429f06a7daebcd18649d6cc2"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:25.351605 11906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "98de37a5a10a45dc889965d52ed26fcd"
format_stamp: "Formatted at 2026-06-21 02:40:25 on dist-test-slave-jkn7"
server_key: "ebfaa73902ef8df8268ec74d68ed411c"
server_key_iv: "f05923a6429f06a7daebcd18649d6cc2"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:25.359891 11906 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.008s
I20260621 02:40:25.377771 11930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:40:25.379608 11906 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:40:25.379798 11906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "98de37a5a10a45dc889965d52ed26fcd"
format_stamp: "Formatted at 2026-06-21 02:40:25 on dist-test-slave-jkn7"
server_key: "ebfaa73902ef8df8268ec74d68ed411c"
server_key_iv: "f05923a6429f06a7daebcd18649d6cc2"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:25.380074 11906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:40:25.400125 11906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:40:25.405340 11906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:40:25.405663 11906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:40:25.406844 11906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:40:25.408679 11906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:40:25.408773 11906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:40:25.408886 11906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:40:25.408964 11906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:40:25.450471 11906 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34195
I20260621 02:40:25.450508 12042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34195 every 8 connection(s)
I20260621 02:40:25.452114 11906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:40:25.454237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 11906
I20260621 02:40:25.454419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:40:25.454994 388 external_mini_cluster.cc:1428] Setting key c1d08d1328c5a7d20ca4ed6742c76b36
Jun 21 02:40:25 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:40:25.478355 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:25.454880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38311 (local address 127.0.97.62:42103)
0621 02:40:25.456681 (+ 1801us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:25.456688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:25.457097 (+ 409us) server_negotiation.cc:407] Connection header received
0621 02:40:25.459861 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:25.459866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:25.459943 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:25.460058 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:40:25.462010 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:25.463597 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:25.465880 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:25.466283 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:25.466420 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:25.470546 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:40:25.470555 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:40:25.470563 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:40:25.470615 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:40:25.474470 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:25.475383 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:25.475392 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:25.475398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:25.475558 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:25.476047 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:25.476054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:25.476060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:25.476463 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:40:25.476621 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:25.477556 (+ 935us) server_negotiation.cc:299] Negotiation successful
0621 02:40:25.477826 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1647,"thread_start_us":1521,"threads_started":1}
I20260621 02:40:25.479398 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:25.455394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42103 (local address 127.0.97.2:38311)
0621 02:40:25.456941 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0621 02:40:25.456986 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:40:25.459549 (+ 2563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:40:25.460308 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:40:25.460321 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:40:25.460947 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:40:25.461821 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:25.461847 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:25.463774 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:25.463782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:25.465727 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:25.465739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:25.466528 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:25.466535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:25.466762 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:25.467480 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:40:25.467517 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:40:25.470271 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:40:25.474626 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:40:25.474636 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:40:25.474654 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:40:25.475233 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:40:25.475684 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:40:25.475688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:40:25.475692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:40:25.475913 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:40:25.476656 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:40:25.476665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:40:25.477070 (+ 405us) client_negotiation.cc:769] Sending connection context
0621 02:40:25.477388 (+ 318us) client_negotiation.cc:240] Negotiation successful
0621 02:40:25.477905 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"thread_start_us":1117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:40:25.481006 12043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42103
I20260621 02:40:25.481349 12043 heartbeater.cc:464] Registering TS with master...
I20260621 02:40:25.482074 12043 heartbeater.cc:511] Master 127.0.97.62:42103 requested a full tablet report, sending...
I20260621 02:40:25.483592 11687 ts_manager.cc:194] Registered new tserver with Master: 98de37a5a10a45dc889965d52ed26fcd (127.0.97.2:34195)
I20260621 02:40:25.485033 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:40:25.513622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37519/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42103
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:40:25.791431 11900 heartbeater.cc:503] Master 127.0.97.62:42103 was elected leader, sending a full tablet report...
W20260621 02:40:25.839437 12049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:40:25.839685 12049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:40:25.839838 12049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:40:25.846882 12049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:40:25.846983 12049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:40:25.847227 12049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:40:25.856905 12049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:37519/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:40:25.858139 12049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:40:25.859398 12049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:40:25.871953 12057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:40:25.872059 12055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:40:25.872341 12054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:40:25.873129 12049 server_base.cc:1034] running on GCE node
I20260621 02:40:25.874181 12049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:40:25.875044 12049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:40:25.876267 12049 hybrid_clock.cc:648] HybridClock initialized: now 1782009625876231 us; error 49 us; skew 500 ppm
I20260621 02:40:25.876566 12049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:40:25 dist-test-slave-jkn7 krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:40:25.879650 12049 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:40:25.880985 12049 webserver.cc:460] Webserver started at http://127.0.97.3:43923/ using document root <none> and password file <none>
I20260621 02:40:25.881701 12049 fs_manager.cc:359] Metadata directory not provided
I20260621 02:40:25.881848 12049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:40:25.882164 12049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:40:25 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:40:25.897156 12049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "77d8242927554eb6a751138075c674bd"
format_stamp: "Formatted at 2026-06-21 02:40:25 on dist-test-slave-jkn7"
server_key: "ed93c3e4956adf5711742407fcda235d"
server_key_iv: "65401985d3d347c08daaf568a63ca724"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:25.898080 12049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "77d8242927554eb6a751138075c674bd"
format_stamp: "Formatted at 2026-06-21 02:40:25 on dist-test-slave-jkn7"
server_key: "ed93c3e4956adf5711742407fcda235d"
server_key_iv: "65401985d3d347c08daaf568a63ca724"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:25.903673 12049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:40:25.920514 12064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:40:25.922097 12049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:40:25.922272 12049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "77d8242927554eb6a751138075c674bd"
format_stamp: "Formatted at 2026-06-21 02:40:25 on dist-test-slave-jkn7"
server_key: "ed93c3e4956adf5711742407fcda235d"
server_key_iv: "65401985d3d347c08daaf568a63ca724"
server_key_version: "kuduclusterkey@0"
I20260621 02:40:25.922528 12049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:40:25.941970 12049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:40:25.945639 12049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:40:25.945931 12049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:40:25.947010 12049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:40:25.948518 12049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:40:25.948592 12049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:40:25.948722 12049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:40:25.948774 12049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:40:25.983855 12049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41695
I20260621 02:40:25.983889 12176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41695 every 8 connection(s)
I20260621 02:40:25.985206 12049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:40:25.995194 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 12049
I20260621 02:40:25.995414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:40:25.995971 388 external_mini_cluster.cc:1428] Setting key c7b9e9cebf40f57d3b5e0e2dd6f00977
Jun 21 02:40:25 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:40:26.005555 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:25.987950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33409 (local address 127.0.97.62:42103)
0621 02:40:25.988424 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:25.988432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:25.989505 (+ 1073us) server_negotiation.cc:407] Connection header received
0621 02:40:25.991165 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:25.991172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:25.991279 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:25.991400 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:40:25.992750 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:25.993593 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:25.995229 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:25.995927 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:25.996067 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.000330 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:40:26.000335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:40:26.000338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:40:26.000378 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:40:26.003039 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.003663 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.003667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.003670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.003790 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.004177 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.004180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.004182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.004409 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:40:26.004529 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.005090 (+ 561us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.005235 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":197,"threads_started":1}
I20260621 02:40:26.006407 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:25.988523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42103 (local address 127.0.97.3:33409)
0621 02:40:25.989346 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0621 02:40:25.989401 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 02:40:25.990893 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:40:25.991601 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:40:25.991609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:40:25.991991 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:40:25.992602 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:25.992619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:25.993732 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:25.993736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:25.995069 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:25.995080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:25.996136 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:25.996141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:25.996374 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:25.997165 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:40:25.997195 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:40:26.000140 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:40:26.003161 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:40:26.003167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:40:26.003179 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:40:26.003563 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:40:26.003916 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:40:26.003920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:40:26.003922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:40:26.004072 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:40:26.004509 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:40:26.004514 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:40:26.004779 (+ 265us) client_negotiation.cc:769] Sending connection context
0621 02:40:26.004997 (+ 218us) client_negotiation.cc:240] Negotiation successful
0621 02:40:26.005402 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":301,"threads_started":1}
I20260621 02:40:26.007637 12177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42103
I20260621 02:40:26.007875 12177 heartbeater.cc:464] Registering TS with master...
I20260621 02:40:26.008549 12177 heartbeater.cc:511] Master 127.0.97.62:42103 requested a full tablet report, sending...
I20260621 02:40:26.009914 11687 ts_manager.cc:194] Registered new tserver with Master: 77d8242927554eb6a751138075c674bd (127.0.97.3:41695)
I20260621 02:40:26.010979 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33409
I20260621 02:40:26.011492 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:40:26.028450 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.016336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.0.97.62:42103)
0621 02:40:26.016477 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.016481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.016962 (+ 481us) server_negotiation.cc:407] Connection header received
0621 02:40:26.017088 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.017091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.017159 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.017236 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:40:26.018547 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.019199 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.020575 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.020966 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.021054 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.023686 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:40:26.023691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:40:26.023694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:40:26.023719 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:40:26.026274 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.026923 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.026927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.026929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.027027 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.027363 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.027368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.027372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.027670 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:40:26.027835 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.028076 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.028233 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:40:26.032219 11687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:40:26.039036 11687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:40:26.062294 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.051435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.0.97.2:34195)
0621 02:40:26.053633 (+ 2198us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.053639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.053661 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:40:26.053734 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.053739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.053976 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.055106 (+ 1130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:40:26.055330 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.056795 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.059747 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.061161 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.061347 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.061579 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.061811 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.061967 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2048,"thread_start_us":1941,"threads_started":1}
I20260621 02:40:26.062286 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.051254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34195 (local address 127.0.0.1:38984)
0621 02:40:26.052141 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0621 02:40:26.052190 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:40:26.052398 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:40:26.054230 (+ 1832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:40:26.054238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:40:26.054267 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:40:26.054802 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.054817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.056924 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.056932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.059572 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.059590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.061381 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.061388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.061565 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.061621 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 02:40:26.061755 (+ 134us) client_negotiation.cc:240] Negotiation successful
0621 02:40:26.061927 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":390,"threads_started":1}
I20260621 02:40:26.063848 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.051410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40913 (local address 127.0.0.1:50510)
0621 02:40:26.052956 (+ 1546us) negotiation.cc:107] Waiting for socket to connect
0621 02:40:26.052979 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:40:26.053109 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:40:26.056353 (+ 3244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:40:26.056358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:40:26.056383 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:40:26.057355 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.057370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.059365 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.059371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.060888 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.060899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.062692 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.062699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.063180 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.063222 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 02:40:26.063311 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 02:40:26.063454 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428,"mutex_wait_us":7,"thread_start_us":1291,"threads_started":1}
I20260621 02:40:26.064234 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.053593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.0.97.3:41695)
0621 02:40:26.054338 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.054348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.055003 (+ 655us) server_negotiation.cc:407] Connection header received
0621 02:40:26.056026 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.056034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.056246 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.056398 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:40:26.057767 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.058894 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.061420 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.062777 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.062893 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.063116 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.063640 (+ 524us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.063803 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":419,"threads_started":1}
I20260621 02:40:26.064596 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.051829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.0.97.1:40913)
0621 02:40:26.055801 (+ 3972us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.055812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.055841 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:40:26.055939 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.055947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.056171 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.056371 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:40:26.057565 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.059200 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.061035 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.062517 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.062645 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.062855 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.064106 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.064262 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3806,"thread_start_us":3663,"threads_started":1}
I20260621 02:40:26.065552 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.053376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41695 (local address 127.0.0.1:44506)
0621 02:40:26.054911 (+ 1535us) negotiation.cc:107] Waiting for socket to connect
0621 02:40:26.054928 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:40:26.055848 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:40:26.057070 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:40:26.057077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:40:26.057102 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:40:26.057600 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.057608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.059067 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.059074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.061235 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.061251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.063265 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.063274 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.063454 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.063482 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 02:40:26.064475 (+ 993us) client_negotiation.cc:240] Negotiation successful
0621 02:40:26.064573 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"thread_start_us":1250,"threads_started":1}
I20260621 02:40:26.066509 11978 tablet_service.cc:1467] Processing CreateTablet for tablet 752fa3b30b294d3f81603473da3ad7ce (DEFAULT_TABLE table=test-table [id=c228490ce7ff48d29ef35daaaa8af7fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:40:26.068009 11978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752fa3b30b294d3f81603473da3ad7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:40:26.069109 12112 tablet_service.cc:1467] Processing CreateTablet for tablet 752fa3b30b294d3f81603473da3ad7ce (DEFAULT_TABLE table=test-table [id=c228490ce7ff48d29ef35daaaa8af7fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:40:26.069542 11835 tablet_service.cc:1467] Processing CreateTablet for tablet 752fa3b30b294d3f81603473da3ad7ce (DEFAULT_TABLE table=test-table [id=c228490ce7ff48d29ef35daaaa8af7fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:40:26.070521 12112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752fa3b30b294d3f81603473da3ad7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:40:26.070711 11835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752fa3b30b294d3f81603473da3ad7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:40:26.082690 12196 tablet_bootstrap.cc:492] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: Bootstrap starting.
I20260621 02:40:26.083459 12197 tablet_bootstrap.cc:492] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: Bootstrap starting.
I20260621 02:40:26.084919 12198 tablet_bootstrap.cc:492] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: Bootstrap starting.
I20260621 02:40:26.087704 12198 tablet_bootstrap.cc:654] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: Neither blocks nor log segments found. Creating new log.
I20260621 02:40:26.087716 12196 tablet_bootstrap.cc:654] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: Neither blocks nor log segments found. Creating new log.
I20260621 02:40:26.087718 12197 tablet_bootstrap.cc:654] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: Neither blocks nor log segments found. Creating new log.
I20260621 02:40:26.088685 12198 log.cc:824] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: Log is configured to *not* fsync() on all Append() calls
I20260621 02:40:26.088894 12196 log.cc:824] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: Log is configured to *not* fsync() on all Append() calls
I20260621 02:40:26.088897 12197 log.cc:824] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: Log is configured to *not* fsync() on all Append() calls
I20260621 02:40:26.092221 12198 tablet_bootstrap.cc:492] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: No bootstrap required, opened a new log
I20260621 02:40:26.092492 12198 ts_tablet_manager.cc:1397] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260621 02:40:26.092983 12196 tablet_bootstrap.cc:492] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: No bootstrap required, opened a new log
I20260621 02:40:26.092983 12197 tablet_bootstrap.cc:492] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: No bootstrap required, opened a new log
I20260621 02:40:26.093248 12197 ts_tablet_manager.cc:1397] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260621 02:40:26.093427 12196 ts_tablet_manager.cc:1397] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260621 02:40:26.100231 12198 raft_consensus.cc:348] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } }
I20260621 02:40:26.100682 12198 raft_consensus.cc:374] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:40:26.100780 12198 raft_consensus.cc:729] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d8242927554eb6a751138075c674bd, State: Initialized, Role: FOLLOWER
I20260621 02:40:26.101543 12198 consensus_queue.cc:260] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } }
I20260621 02:40:26.102654 12177 heartbeater.cc:503] Master 127.0.97.62:42103 was elected leader, sending a full tablet report...
I20260621 02:40:26.103338 12198 ts_tablet_manager.cc:1428] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260621 02:40:26.104373 12197 raft_consensus.cc:348] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } }
I20260621 02:40:26.104805 12197 raft_consensus.cc:374] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:40:26.104897 12197 raft_consensus.cc:729] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98de37a5a10a45dc889965d52ed26fcd, State: Initialized, Role: FOLLOWER
I20260621 02:40:26.104780 12196 raft_consensus.cc:348] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } }
I20260621 02:40:26.105180 12196 raft_consensus.cc:374] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:40:26.105307 12196 raft_consensus.cc:729] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac6b10cbe734b56a40865d6c28dd760, State: Initialized, Role: FOLLOWER
I20260621 02:40:26.105623 12197 consensus_queue.cc:260] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } }
I20260621 02:40:26.105844 12196 consensus_queue.cc:260] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } }
I20260621 02:40:26.107897 12196 ts_tablet_manager.cc:1428] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260621 02:40:26.108879 12043 heartbeater.cc:503] Master 127.0.97.62:42103 was elected leader, sending a full tablet report...
I20260621 02:40:26.109474 12197 ts_tablet_manager.cc:1428] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260621 02:40:26.204142 12044 tablet.cc:2369] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:40:26.237715 12178 tablet.cc:2369] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:40:26.238153 12203 raft_consensus.cc:482] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:40:26.238456 12203 raft_consensus.cc:504] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } }
I20260621 02:40:26.240142 12203 leader_election.cc:288] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac6b10cbe734b56a40865d6c28dd760 (127.0.97.1:40913), 77d8242927554eb6a751138075c674bd (127.0.97.3:41695)
I20260621 02:40:26.249361 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.240484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44687 (local address 127.0.97.3:41695)
0621 02:40:26.240706 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.240716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.241871 (+ 1155us) server_negotiation.cc:407] Connection header received
0621 02:40:26.242141 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.242145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.242201 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.242268 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:40:26.242796 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.243731 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.246764 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.248040 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.248179 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.248282 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.248962 (+ 680us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.249099 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:40:26.249368 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.241171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39913 (local address 127.0.97.1:40913)
0621 02:40:26.241395 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.241400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.241420 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:40:26.241575 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.241580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.241684 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.241783 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:40:26.242400 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.243442 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.246769 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.248056 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.248179 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.248282 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.248962 (+ 680us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.249093 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:40:26.249599 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.240436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41695 (local address 127.0.97.2:44687)
0621 02:40:26.241774 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0621 02:40:26.241796 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:40:26.241991 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:40:26.242312 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:40:26.242319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:40:26.242348 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:40:26.242641 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.242648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.244240 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.244246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.246578 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.246592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.248353 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.248359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.248649 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.248807 (+ 158us) client_negotiation.cc:769] Sending connection context
0621 02:40:26.248906 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 02:40:26.249087 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":248,"thread_start_us":613,"threads_started":1}
I20260621 02:40:26.249630 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.240415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40913 (local address 127.0.97.2:39913)
0621 02:40:26.241234 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0621 02:40:26.241257 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:40:26.241409 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:40:26.241824 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:40:26.241832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:40:26.241865 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:40:26.242216 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.242225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.243585 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.243589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.246582 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:40:26.246592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.248356 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.248361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:40:26.248651 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.248806 (+ 155us) client_negotiation.cc:769] Sending connection context
0621 02:40:26.248906 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:40:26.249087 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":239,"thread_start_us":468,"threads_started":1}
I20260621 02:40:26.250679 12132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752fa3b30b294d3f81603473da3ad7ce" candidate_uuid: "98de37a5a10a45dc889965d52ed26fcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d8242927554eb6a751138075c674bd" is_pre_election: true
I20260621 02:40:26.250672 11855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752fa3b30b294d3f81603473da3ad7ce" candidate_uuid: "98de37a5a10a45dc889965d52ed26fcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac6b10cbe734b56a40865d6c28dd760" is_pre_election: true
I20260621 02:40:26.251230 11855 raft_consensus.cc:2454] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98de37a5a10a45dc889965d52ed26fcd in term 0.
I20260621 02:40:26.251230 12132 raft_consensus.cc:2454] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98de37a5a10a45dc889965d52ed26fcd in term 0.
I20260621 02:40:26.251938 11931 leader_election.cc:302] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77d8242927554eb6a751138075c674bd, 98de37a5a10a45dc889965d52ed26fcd; no voters:
I20260621 02:40:26.252348 12203 raft_consensus.cc:2781] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:40:26.252514 12203 raft_consensus.cc:482] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:40:26.252624 12203 raft_consensus.cc:3037] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:40:26.254849 12203 raft_consensus.cc:504] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } }
I20260621 02:40:26.255429 12203 leader_election.cc:288] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [CANDIDATE]: Term 1 election: Requested vote from peers 3ac6b10cbe734b56a40865d6c28dd760 (127.0.97.1:40913), 77d8242927554eb6a751138075c674bd (127.0.97.3:41695)
I20260621 02:40:26.255934 11855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752fa3b30b294d3f81603473da3ad7ce" candidate_uuid: "98de37a5a10a45dc889965d52ed26fcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac6b10cbe734b56a40865d6c28dd760"
I20260621 02:40:26.256177 11855 raft_consensus.cc:3037] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [term 0 FOLLOWER]: Advancing to term 1
W20260621 02:40:26.256223 11904 tablet.cc:2369] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:40:26.258543 11855 raft_consensus.cc:2454] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98de37a5a10a45dc889965d52ed26fcd in term 1.
I20260621 02:40:26.258944 11933 leader_election.cc:302] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac6b10cbe734b56a40865d6c28dd760, 98de37a5a10a45dc889965d52ed26fcd; no voters:
I20260621 02:40:26.258896 12132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752fa3b30b294d3f81603473da3ad7ce" candidate_uuid: "98de37a5a10a45dc889965d52ed26fcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d8242927554eb6a751138075c674bd"
I20260621 02:40:26.259123 12132 raft_consensus.cc:3037] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:40:26.259689 12203 raft_consensus.cc:2781] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:40:26.260210 12203 raft_consensus.cc:686] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 1 LEADER]: Becoming Leader. State: Replica: 98de37a5a10a45dc889965d52ed26fcd, State: Running, Role: LEADER
I20260621 02:40:26.260660 12203 consensus_queue.cc:237] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } }
I20260621 02:40:26.262001 12132 raft_consensus.cc:2454] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98de37a5a10a45dc889965d52ed26fcd in term 1.
I20260621 02:40:26.265018 11686 catalog_manager.cc:5495] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 98de37a5a10a45dc889965d52ed26fcd (127.0.97.2). New cstate: current_term: 1 leader_uuid: "98de37a5a10a45dc889965d52ed26fcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98de37a5a10a45dc889965d52ed26fcd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 } health_report { overall_health: UNKNOWN } } }
I20260621 02:40:26.299443 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.294225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.0.97.2:34195)
0621 02:40:26.294432 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.294437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.294456 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:40:26.294603 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.294607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.294699 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.294852 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:40:26.295213 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.295918 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.297341 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.297733 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.297871 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.298368 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:40:26.298856 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:40:26.299013 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.299094 (+ 81us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.299168 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:40:26.307801 12132 raft_consensus.cc:1261] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [term 1 FOLLOWER]: Refusing update from remote peer 98de37a5a10a45dc889965d52ed26fcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:40:26.307843 11855 raft_consensus.cc:1261] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [term 1 FOLLOWER]: Refusing update from remote peer 98de37a5a10a45dc889965d52ed26fcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:40:26.308781 12207 consensus_queue.cc:1035] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "77d8242927554eb6a751138075c674bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:40:26.309103 12203 consensus_queue.cc:1035] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac6b10cbe734b56a40865d6c28dd760" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:40:26.319377 12210 mvcc.cc:205] Tried to move back new op lower bound from 7299111429346885632 to 7299111429167427584. Current Snapshot: MvccSnapshot[applied={T|T < 7299111429346885632}]
I20260621 02:40:26.340122 11686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:40:26.340408 11686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:40:26.343955 11686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e66522c9dec24afb81f31b1f7d61afc4: Sending DeleteTablet for 3 replicas of tablet 752fa3b30b294d3f81603473da3ad7ce
I20260621 02:40:26.345078 11978 tablet_service.cc:1514] Processing DeleteTablet for tablet 752fa3b30b294d3f81603473da3ad7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:40:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38984
I20260621 02:40:26.345155 11835 tablet_service.cc:1514] Processing DeleteTablet for tablet 752fa3b30b294d3f81603473da3ad7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:40:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260621 02:40:26.345412 12112 tablet_service.cc:1514] Processing DeleteTablet for tablet 752fa3b30b294d3f81603473da3ad7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:40:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44506
I20260621 02:40:26.348281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 11767
I20260621 02:40:26.355631 12222 tablet_replica.cc:330] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: stopping tablet replica
I20260621 02:40:26.356134 12222 raft_consensus.cc:2229] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:40:26.356683 12222 raft_consensus.cc:2258] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:40:26.357812 12221 tablet_replica.cc:330] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: stopping tablet replica
I20260621 02:40:26.358263 12221 raft_consensus.cc:2229] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:40:26.358783 12221 raft_consensus.cc:2258] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:40:26.360678 12222 ts_tablet_manager.cc:1905] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:40:26.367986 12222 ts_tablet_manager.cc:1918] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:40:26.368144 12222 log.cc:1196] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/752fa3b30b294d3f81603473da3ad7ce
I20260621 02:40:26.368634 12222 ts_tablet_manager.cc:1939] T 752fa3b30b294d3f81603473da3ad7ce P 77d8242927554eb6a751138075c674bd: Deleting consensus metadata
I20260621 02:40:26.371411 12221 ts_tablet_manager.cc:1905] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:40:26.373817 12220 tablet_replica.cc:330] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: stopping tablet replica
Jun 21 02:40:26 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:40:26.374841 11671 catalog_manager.cc:4841] TS 77d8242927554eb6a751138075c674bd (127.0.97.3:41695): tablet 752fa3b30b294d3f81603473da3ad7ce (table test-table [id=c228490ce7ff48d29ef35daaaa8af7fc]) successfully deleted
I20260621 02:40:26.375617 12220 raft_consensus.cc:2229] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:40:26.376380 12220 raft_consensus.cc:2258] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:40:26.378708 12220 ts_tablet_manager.cc:1905] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:40:26.379153 12221 ts_tablet_manager.cc:1918] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:40:26.379338 12221 log.cc:1196] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/752fa3b30b294d3f81603473da3ad7ce
I20260621 02:40:26.379869 12221 ts_tablet_manager.cc:1939] T 752fa3b30b294d3f81603473da3ad7ce P 3ac6b10cbe734b56a40865d6c28dd760: Deleting consensus metadata
I20260621 02:40:26.381708 11673 catalog_manager.cc:4841] TS 3ac6b10cbe734b56a40865d6c28dd760 (127.0.97.1:40913): tablet 752fa3b30b294d3f81603473da3ad7ce (table test-table [id=c228490ce7ff48d29ef35daaaa8af7fc]) successfully deleted
I20260621 02:40:26.383765 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.349570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.0.97.1:40913)
0621 02:40:26.349783 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.349792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.363166 (+ 13374us) server_negotiation.cc:407] Connection header received
0621 02:40:26.363598 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.363603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.363705 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.363779 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:40:26.365604 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.366742 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.369029 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.369703 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.369796 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.376053 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:40:26.376069 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:40:26.376086 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:40:26.376139 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:40:26.380613 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.381551 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.381560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.381563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.381721 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.382183 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.382187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.382189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.382522 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:40:26.382832 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.383155 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.383364 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:40:26.384529 11895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:40:26.386405 12220 ts_tablet_manager.cc:1918] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:40:26.387091 12220 log.cc:1196] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/752fa3b30b294d3f81603473da3ad7ce
I20260621 02:40:26.387712 12220 ts_tablet_manager.cc:1939] T 752fa3b30b294d3f81603473da3ad7ce P 98de37a5a10a45dc889965d52ed26fcd: Deleting consensus metadata
I20260621 02:40:26.389374 11674 catalog_manager.cc:4841] TS 98de37a5a10a45dc889965d52ed26fcd (127.0.97.2:34195): tablet 752fa3b30b294d3f81603473da3ad7ce (table test-table [id=c228490ce7ff48d29ef35daaaa8af7fc]) successfully deleted
I20260621 02:40:26.474926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 11767
I20260621 02:40:26.492929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 11906
Jun 21 02:40:26 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:40:26.510600 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.493781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.0.97.2:34195)
0621 02:40:26.494029 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.494035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.494056 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:40:26.494226 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.494233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.494317 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.494437 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:40:26.495775 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.497031 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.498439 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.499135 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.499259 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.503739 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:40:26.503755 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:40:26.503775 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:40:26.503825 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:40:26.507752 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.508404 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.508414 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.508418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.508586 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.508955 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.508960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.508964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.509475 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:40:26.509800 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.509984 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.510215 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:40:26.511399 12038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:40:26.622000 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 11906
I20260621 02:40:26.640882 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 12049
Jun 21 02:40:26 dist-test-slave-jkn7 krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:40:26.655407 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:40:26.641677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.0.97.3:41695)
0621 02:40:26.641846 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 02:40:26.641851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:40:26.641954 (+ 103us) server_negotiation.cc:407] Connection header received
0621 02:40:26.642058 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:40:26.642062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:40:26.642140 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:40:26.642228 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:40:26.643399 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.644117 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.645478 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:40:26.645864 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:40:26.645978 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:40:26.650136 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:40:26.650151 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:40:26.650171 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:40:26.650206 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:40:26.652810 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.653619 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.653626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.653629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.653755 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:40:26.654130 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:40:26.654133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:40:26.654135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:40:26.654500 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:40:26.654712 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 02:40:26.654979 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 02:40:26.655153 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:40:26.656065 12172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:40:26.745182 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 12049
I20260621 02:40:26.762077 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 11655
I20260621 02:40:26.763234 11717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:40:26.874118 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 11655
I20260621 02:40:26.896454 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 02:40:27.231201 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 02:40:27.231395 388 mini_ranger.cc:66] Stopping Ranger...
I20260621 02:40:27.572491 388 mini_ranger.cc:68] Stopped Ranger
2026-06-21 02:40:27.572 UTC [11241] LOG: received smart shutdown request
2026-06-21 02:40:27.576 UTC [11241] LOG: background worker "logical replication launcher" (PID 11250) exited with exit code 1
2026-06-21 02:40:27.576 UTC [11245] LOG: shutting down
2026-06-21 02:40:27.730 UTC [11241] LOG: database system is shut down
2026-06-21T02:40:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48404 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:40:27 dist-test-slave-jkn7 krb5kdc[12236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:40:27 dist-test-slave-jkn7 krb5kdc[12236](info): set up 2 sockets
Jun 21 02:40:27 dist-test-slave-jkn7 krb5kdc[12236](info): commencing operation
krb5kdc: starting...
W20260621 02:40:28.953552 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:40:29 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:40:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260621 02:40:29.253937 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskXxBWLs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/postgres -l logfile start
2026-06-21 02:40:31.635 UTC [12266] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 02:40:31.635 UTC [12266] LOG: listening on IPv4 address "127.0.97.20", port 40805
2026-06-21 02:40:31.638 UTC [12266] LOG: listening on Unix socket "/tmp/.s.PGSQL.40805"
2026-06-21 02:40:31.659 UTC [12269] LOG: database system was shut down at 2026-06-21 02:40:30 UTC
2026-06-21 02:40:31.663 UTC [12266] LOG: database system is ready to accept connections
I20260621 02:40:32.633268 388 mini_postgres.cc:96] Postgres bound to 40805
2026-06-21 02:40:32.650 UTC [12277] FATAL: database "slave" does not exist
127.0.97.20:40805 - accepting connections
I20260621 02:40:32.650734 388 mini_ranger.cc:161] Starting Ranger...
I20260621 02:40:32.679507 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 02:40:33.132987 388 mini_ranger.cc:87] Created ranger Postgres database
I20260621 02:40:33.133114 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-admin
2026-06-21 02:40:33,423 [I] DB FLAVOR :POSTGRES
2026-06-21 02:40:33,424 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:40:33,424 [I] Checking connection..
2026-06-21 02:40:33,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 02:40:33,659 [I] Checking connection passed.
2026-06-21 02:40:33,659 [I] --------- Verifying version history table ---------
2026-06-21 02:40:33,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:40:33,891 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:40:33,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:40:34,142 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:40:34,142 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 02:40:34,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 02:40:34.382 UTC [12341] WARNING: there is no transaction in progress
2026-06-21 02:40:34,388 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 02:40:34,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:40:34,637 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 02:40:34,637 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:40:34,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 02:40:34,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 02:40:35,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jkn7.c.gcp-upstream.internal','N') ;"
2026-06-21 02:40:35,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:40:35,627 [I] Table x_portal_user does not exist in database ranger
2026-06-21 02:40:35,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:40:35,881 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 02:40:35,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:40:36,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:40:36,340 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 02:40:36,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 02:40:37.239 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.254 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.269 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.284 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.301 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.318 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.375 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.382 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.398 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.408 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.417 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.431 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.440 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.446 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.452 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:37.459 UTC [12449] WARNING: there is no transaction in progress
2026-06-21 02:40:38,071 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 02:40:38,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 02:40:38,309 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 02:40:38,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:40:38,545 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 02:40:38,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:40:38,803 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 02:40:38,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:40:39,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:40:39,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jkn7.c.gcp-upstream.internal';"
2026-06-21 02:40:39,516 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 02:40:39,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 02:40:39,772 [I] DB_PATCHES have already been applied
I20260621 02:40:39.779743 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskXxBWLs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 02:40:39.779842 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260621 02:40:39.796362 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:42025
Jun 21, 2026 2:40:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:40:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:40:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 2:40:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 2:40:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 2:40:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 2:40:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 02:40:40.942874 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.292s user 0.003s sys 0.040s
Jun 21, 2026 2:41:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 2:41:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 2:41:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 2:41:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 2:41:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 02:41:01.632624 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 02:41:01 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009661, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 02:41:01.804534 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 02:41:01.825445 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 02:41:02.649431 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 02:41:02.649524 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-kms
I20260621 02:41:02.649578 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:40805
2026-06-21 02:41:03,000 [I] DB FLAVOR :POSTGRES
2026-06-21 02:41:03,000 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:41:03,000 [I] Checking connection
2026-06-21 02:41:03,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 02:41:03,237 [I] connection success
2026-06-21 02:41:03,237 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 02:41:03,238 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 02:41:03,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 02:41:03,476 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 02:41:03,477 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:41:03,477 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 02:41:03,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 02:41:03,743 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 02:41:04.236100 388 mini_ranger_kms.cc:310] Created kms service
I20260621 02:41:04.444766 388 mini_ranger_kms.cc:326] Created kudu user
I20260621 02:41:04.489065 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 02:41:04 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009661, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:04.870119 388 mini_ranger_kms.cc:384] Added ranger policy
I20260621 02:41:04.870445 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskXxBWLs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskXxBWLs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 02:41:04.870496 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260621 02:41:04.888513 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12629
I20260621 02:41:04.888661 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:40545
Jun 21, 2026 2:41:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:41:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:41:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 2:41:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 2:41:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 2:41:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 02:41:07.185169 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.381s user 0.078s sys 0.283s
I20260621 02:41:07.185529 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 02:41:07.185710 388 mini_ranger_kms.cc:401] 127.0.97.20:40545/kms/v1/keys
Jun 21 02:41:07 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782009667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 02:41:07 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 21 02:41:08 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 21 02:41:10 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:41:10.356228 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42379
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42379
--ranger_config_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40545/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:10.709348 12680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:10.709645 12680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:10.709801 12680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:10.716790 12680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:10.716883 12680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:10.716982 12680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:10.717088 12680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:10.717173 12680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:10.726893 12680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:40545/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42379
--ranger_config_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:10.728058 12680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:10.729346 12680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:10.740772 12686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:10.740744 12688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:10.740919 12685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:10.741463 12680 server_base.cc:1034] running on GCE node
I20260621 02:41:10.742266 12680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:10.743572 12680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:10.744771 12680 hybrid_clock.cc:648] HybridClock initialized: now 1782009670744734 us; error 32 us; skew 500 ppm
I20260621 02:41:10.745116 12680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:10 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:10.751313 12680 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:41:10.752748 12680 webserver.cc:460] Webserver started at http://127.0.97.62:34555/ using document root <none> and password file <none>
I20260621 02:41:10.753701 12680 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:10.753806 12680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:10.754076 12680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:41:10 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:10.803411 12680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "88aeaba99f2e44bbb7047e5f9b275f68"
format_stamp: "Formatted at 2026-06-21 02:41:10 on dist-test-slave-jkn7"
server_key: "d04f060f70edc43193c030074b940ae8"
server_key_iv: "0d54442cd85c6803c4f7d0f12ae3d560"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:10.804260 12680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "88aeaba99f2e44bbb7047e5f9b275f68"
format_stamp: "Formatted at 2026-06-21 02:41:10 on dist-test-slave-jkn7"
server_key: "d04f060f70edc43193c030074b940ae8"
server_key_iv: "0d54442cd85c6803c4f7d0f12ae3d560"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:10.809679 12680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:41:10.825450 12695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:10.826999 12680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:41:10.827185 12680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "88aeaba99f2e44bbb7047e5f9b275f68"
format_stamp: "Formatted at 2026-06-21 02:41:10 on dist-test-slave-jkn7"
server_key: "d04f060f70edc43193c030074b940ae8"
server_key_iv: "0d54442cd85c6803c4f7d0f12ae3d560"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:10.827404 12680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:10.860605 12680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:10.864545 12680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:10.864897 12680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:10.883666 12680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42379
I20260621 02:41:10.883687 12746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42379 every 8 connection(s)
I20260621 02:41:10.885272 12680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:41:10.885671 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 12680
I20260621 02:41:10.885890 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:41:10.886732 388 external_mini_cluster.cc:1428] Setting key fa652c255ac7ee1bb9ea1a2d61be20c2
I20260621 02:41:10.889979 12747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:41:10 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:10.903021 12747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Bootstrap starting.
I20260621 02:41:10.907141 12747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:10.908432 12747 log.cc:824] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:10.910957 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:10.888843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.0.97.62:42379)
0621 02:41:10.889667 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:10.889682 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:10.889731 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:41:10.892018 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:10.892053 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:10.892548 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:10.893017 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:10.894337 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:10.896452 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:10.897795 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:10.898832 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:10.899015 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:10.902762 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:10.902773 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:10.902803 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:10.902852 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:10.906564 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:10.907206 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:10.907215 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:10.907223 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:10.907409 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:10.907778 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:10.907783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:10.907786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:10.908438 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:10.908736 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:10.909233 (+ 497us) server_negotiation.cc:299] Negotiation successful
0621 02:41:10.909696 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":391,"threads_started":1}
I20260621 02:41:10.911886 12747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: No bootstrap required, opened a new log
I20260621 02:41:10.919463 12747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } }
I20260621 02:41:10.919806 12747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:10.919873 12747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88aeaba99f2e44bbb7047e5f9b275f68, State: Initialized, Role: FOLLOWER
I20260621 02:41:10.920363 12747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } }
I20260621 02:41:10.920557 12747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:41:10.920652 12747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:41:10.920784 12747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:10.923087 12747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } }
I20260621 02:41:10.923549 12747 leader_election.cc:302] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88aeaba99f2e44bbb7047e5f9b275f68; no voters:
I20260621 02:41:10.924108 12747 leader_election.cc:288] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:41:10.924259 12752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:41:10.925005 12752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 1 LEADER]: Becoming Leader. State: Replica: 88aeaba99f2e44bbb7047e5f9b275f68, State: Running, Role: LEADER
I20260621 02:41:10.925590 12752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } }
I20260621 02:41:10.926255 12747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:41:10.928613 12754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88aeaba99f2e44bbb7047e5f9b275f68. Latest consensus state: current_term: 1 leader_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } } }
I20260621 02:41:10.928983 12754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:10.928722 12753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } } }
I20260621 02:41:10.929116 12753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:10.931257 12760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:41:10.936205 12760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:41:10.944762 12747 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 02:41:10.945955 12760 catalog_manager.cc:1347] Generated new cluster ID: 2d2089637d574c8ea4913229712187a4
I20260621 02:41:10.946076 12760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:41:10.987843 12760 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:41:10.989934 12760 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:41:11.005505 12760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Generated new TSK 0
I20260621 02:41:11.006678 12760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:41:11.729020 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42379
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:41:12 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782009672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 02:41:12.133843 12792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:12.134323 12792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:12.134531 12792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:12.146248 12792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:12.146378 12792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:12.146618 12792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:41:12.162106 12792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:40545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:12.163734 12792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:12.165369 12792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:12.183015 12802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:12.183583 12803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:12.191982 12792 server_base.cc:1034] running on GCE node
W20260621 02:41:12.192229 12805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:12.193034 12792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:12.194312 12792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:12.195596 12792 hybrid_clock.cc:648] HybridClock initialized: now 1782009672195520 us; error 63 us; skew 500 ppm
I20260621 02:41:12.195996 12792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:12 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:12.200172 12792 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:41:12.205705 12792 webserver.cc:460] Webserver started at http://127.0.97.1:37873/ using document root <none> and password file <none>
I20260621 02:41:12.206681 12792 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:12.206826 12792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:12.207201 12792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:41:12 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:12.228504 12792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "1651ca89d9604dd7b83fc204295b499f"
format_stamp: "Formatted at 2026-06-21 02:41:12 on dist-test-slave-jkn7"
server_key: "7b86ab1f3306fef104ee3464c2ea6044"
server_key_iv: "85c8c2aae4b38ebf58a582cbc53b3220"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:12.229733 12792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1651ca89d9604dd7b83fc204295b499f"
format_stamp: "Formatted at 2026-06-21 02:41:12 on dist-test-slave-jkn7"
server_key: "7b86ab1f3306fef104ee3464c2ea6044"
server_key_iv: "85c8c2aae4b38ebf58a582cbc53b3220"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:12.236850 12792 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.005s
I20260621 02:41:12.269898 12812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:12.272010 12792 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.007s sys 0.008s
I20260621 02:41:12.272296 12792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "1651ca89d9604dd7b83fc204295b499f"
format_stamp: "Formatted at 2026-06-21 02:41:12 on dist-test-slave-jkn7"
server_key: "7b86ab1f3306fef104ee3464c2ea6044"
server_key_iv: "85c8c2aae4b38ebf58a582cbc53b3220"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:12.272652 12792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:12.308399 12792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:12.312258 12792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:12.312563 12792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:12.313530 12792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:12.314863 12792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:12.314936 12792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:12.315095 12792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:12.315156 12792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:12.378016 12792 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45223
I20260621 02:41:12.378471 12924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45223 every 8 connection(s)
I20260621 02:41:12.379737 12792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:41:12.390054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 12792
I20260621 02:41:12.390259 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:41:12.391083 388 external_mini_cluster.cc:1428] Setting key 51ac8135192cd4db2ec41e4ee8c04a6e
Jun 21 02:41:12 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260621 02:41:12.423219 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:12.383356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42379 (local address 127.0.97.1:51607)
0621 02:41:12.389924 (+ 6568us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:12.389986 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:12.392725 (+ 2739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:12.400809 (+ 8084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:12.400829 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:12.401545 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:12.402389 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:12.402417 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:12.404122 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:12.404130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:12.406396 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:12.406415 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:12.407790 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:12.407799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:12.408112 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:12.408941 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:12.408974 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:12.411933 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:12.417705 (+ 5772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:12.417717 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:12.417735 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:12.418334 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:12.418961 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:12.418967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:12.418973 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:12.419229 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:12.420017 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:12.420025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:12.420635 (+ 610us) client_negotiation.cc:769] Sending connection context
0621 02:41:12.420961 (+ 326us) client_negotiation.cc:240] Negotiation successful
0621 02:41:12.421692 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6209,"thread_start_us":9150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260621 02:41:12.428493 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:12.382786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51607 (local address 127.0.97.62:42379)
0621 02:41:12.400209 (+ 17423us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:12.400217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:12.400239 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:41:12.400301 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:12.400306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:12.400393 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:12.400522 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:12.402641 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:12.403935 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:12.406625 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:12.407357 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:12.407498 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:12.412208 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:12.412217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:12.412226 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:12.412280 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:12.417405 (+ 5125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:12.418635 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:12.418642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:12.418647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:12.418809 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:12.419375 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:12.419381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:12.419385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:12.419787 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:12.419944 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:12.427846 (+ 7902us) server_negotiation.cc:299] Negotiation successful
0621 02:41:12.428112 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17290,"thread_start_us":17849,"threads_started":1}
I20260621 02:41:12.430493 12925 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42379
I20260621 02:41:12.430800 12925 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:12.431746 12925 heartbeater.cc:511] Master 127.0.97.62:42379 requested a full tablet report, sending...
I20260621 02:41:12.434340 12712 ts_manager.cc:194] Registered new tserver with Master: 1651ca89d9604dd7b83fc204295b499f (127.0.97.1:45223)
I20260621 02:41:12.436028 12712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:41:12.458662 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42379
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:41:12 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 21 02:41:12 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260621 02:41:12.913849 12931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:12.914290 12931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:12.914489 12931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:12.925659 12931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:12.925869 12931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:12.926087 12931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:41:12.941704 12931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:40545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:12.943665 12931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:12.945389 12931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:12.993162 12939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:13.007858 12938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:13.021835 12931 server_base.cc:1034] running on GCE node
W20260621 02:41:13.023698 12941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:13.025807 12931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:13.037569 12931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:13.041021 12931 hybrid_clock.cc:648] HybridClock initialized: now 1782009673040903 us; error 111 us; skew 500 ppm
I20260621 02:41:13.041491 12931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:13 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:13.045850 12931 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:41:13.092394 12931 webserver.cc:460] Webserver started at http://127.0.97.2:34871/ using document root <none> and password file <none>
I20260621 02:41:13.093565 12931 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:13.093765 12931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:13.094190 12931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:41:13 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:13.112377 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "678b5adb2aae4c68acce1feaa80da6d4"
format_stamp: "Formatted at 2026-06-21 02:41:13 on dist-test-slave-jkn7"
server_key: "3c820b7dd29338c406e9079a71868c76"
server_key_iv: "558faa01178917f5fc69a4a34c50946b"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:13.113683 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "678b5adb2aae4c68acce1feaa80da6d4"
format_stamp: "Formatted at 2026-06-21 02:41:13 on dist-test-slave-jkn7"
server_key: "3c820b7dd29338c406e9079a71868c76"
server_key_iv: "558faa01178917f5fc69a4a34c50946b"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:13.136135 12931 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.009s sys 0.008s
I20260621 02:41:13.169850 12948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:13.171895 12931 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.010s sys 0.007s
I20260621 02:41:13.172176 12931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "678b5adb2aae4c68acce1feaa80da6d4"
format_stamp: "Formatted at 2026-06-21 02:41:13 on dist-test-slave-jkn7"
server_key: "3c820b7dd29338c406e9079a71868c76"
server_key_iv: "558faa01178917f5fc69a4a34c50946b"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:13.172536 12931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:13.253664 12931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:13.258832 12931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:13.259205 12931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:13.266505 12931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:13.274487 12931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:13.274646 12931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260621 02:41:13.274807 12931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:13.274895 12931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:13.432211 12931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40621
I20260621 02:41:13.432256 13067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40621 every 8 connection(s)
I20260621 02:41:13.433851 12931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:41:13.439723 12925 heartbeater.cc:503] Master 127.0.97.62:42379 was elected leader, sending a full tablet report...
I20260621 02:41:13.441731 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 12931
I20260621 02:41:13.441944 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:41:13.442659 388 external_mini_cluster.cc:1428] Setting key 16a82157f8b912ee2cc32db05baca65c
Jun 21 02:41:13 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:13.473757 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:13.436864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37663 (local address 127.0.97.62:42379)
0621 02:41:13.449833 (+ 12969us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:13.449840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:13.450851 (+ 1011us) server_negotiation.cc:407] Connection header received
0621 02:41:13.453723 (+ 2872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:13.453731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:13.453822 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:13.453902 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:13.456242 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:13.457433 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:13.460017 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:13.460734 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:13.460859 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:13.465540 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:13.465549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:13.465557 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:13.465601 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:13.469509 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:13.470557 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:13.470565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:13.470572 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:13.470729 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:13.471297 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:13.471305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:13.471311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:13.471716 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:13.471864 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:13.472937 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0621 02:41:13.473221 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12851,"thread_start_us":12737,"threads_started":1}
I20260621 02:41:13.476271 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:13.437422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42379 (local address 127.0.97.2:37663)
0621 02:41:13.450438 (+ 13016us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:13.450493 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:13.453394 (+ 2901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:13.454463 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:13.454479 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:13.455159 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:13.456016 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:13.456057 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:13.457651 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:13.457660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:13.459849 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:13.459865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:13.461065 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:13.461075 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:13.461417 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:13.462227 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:13.462262 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:13.465258 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:13.469726 (+ 4468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:13.469738 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:13.469760 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:13.470405 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:13.470883 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:13.470891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:13.470897 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:13.471160 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:13.471973 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:13.471984 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:13.472447 (+ 463us) client_negotiation.cc:769] Sending connection context
0621 02:41:13.474188 (+ 1741us) client_negotiation.cc:240] Negotiation successful
0621 02:41:13.474785 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12701,"thread_start_us":12472,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:41:13.478096 13068 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42379
I20260621 02:41:13.478415 13068 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:13.479271 13068 heartbeater.cc:511] Master 127.0.97.62:42379 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260621 02:41:13.480935 12712 ts_manager.cc:194] Registered new tserver with Master: 678b5adb2aae4c68acce1feaa80da6d4 (127.0.97.2:40621)
I20260621 02:41:13.482205 12712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:41:13.507686 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42379
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:13.837636 13074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:13.837903 13074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:13.838095 13074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:13.845270 13074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:13.845419 13074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:13.845600 13074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:41:13.855569 13074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:40545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:13.856843 13074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:13.858212 13074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:13.870127 13079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:13.870231 13080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:13.870400 13082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:13.871291 13074 server_base.cc:1034] running on GCE node
I20260621 02:41:13.871867 13074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:13.872735 13074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:13.873966 13074 hybrid_clock.cc:648] HybridClock initialized: now 1782009673873928 us; error 46 us; skew 500 ppm
I20260621 02:41:13.874292 13074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:13 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:13.877405 13074 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:41:13.878885 13074 webserver.cc:460] Webserver started at http://127.0.97.3:33547/ using document root <none> and password file <none>
I20260621 02:41:13.879561 13074 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:13.879706 13074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:13.879995 13074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:41:13 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:13.898175 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "577f571fdb2746b6ad55ad9c476b1ff9"
format_stamp: "Formatted at 2026-06-21 02:41:13 on dist-test-slave-jkn7"
server_key: "35cd81b3661dad4fb14bce94ac6585ca"
server_key_iv: "f279128eb5e77ce184212fcc6b252bce"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:13.899242 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "577f571fdb2746b6ad55ad9c476b1ff9"
format_stamp: "Formatted at 2026-06-21 02:41:13 on dist-test-slave-jkn7"
server_key: "35cd81b3661dad4fb14bce94ac6585ca"
server_key_iv: "f279128eb5e77ce184212fcc6b252bce"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:13.906049 13074 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260621 02:41:13.923997 13089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:13.925832 13074 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:41:13.926056 13074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "577f571fdb2746b6ad55ad9c476b1ff9"
format_stamp: "Formatted at 2026-06-21 02:41:13 on dist-test-slave-jkn7"
server_key: "35cd81b3661dad4fb14bce94ac6585ca"
server_key_iv: "f279128eb5e77ce184212fcc6b252bce"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:13.926329 13074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:13.956776 13074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:13.960462 13074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:13.960719 13074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:13.961642 13074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:13.962924 13074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:13.963009 13074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:13.963094 13074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:13.963128 13074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:13.997713 13074 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40303
I20260621 02:41:13.997761 13201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40303 every 8 connection(s)
I20260621 02:41:13.998966 13074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:41:13.999893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 13074
I20260621 02:41:14.000094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:41:14.000648 388 external_mini_cluster.cc:1428] Setting key 1fe7ab994c3787659b61e4be864fafe0
Jun 21 02:41:14 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:14.017936 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:14.001785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40951 (local address 127.0.97.62:42379)
0621 02:41:14.002443 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:14.002449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:14.005148 (+ 2699us) server_negotiation.cc:407] Connection header received
0621 02:41:14.005196 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:14.005199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:14.005270 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:14.005429 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:14.006742 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.007427 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.008868 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.009328 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.009438 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:14.012411 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:14.012416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:14.012419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:14.012448 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:14.015065 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:14.015773 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:14.015780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:14.015783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:14.015909 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:14.016341 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:14.016348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:14.016351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:14.016599 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:14.016741 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:14.017370 (+ 629us) server_negotiation.cc:299] Negotiation successful
0621 02:41:14.017582 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":348,"threads_started":1}
I20260621 02:41:14.018760 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:14.002369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42379 (local address 127.0.97.3:40951)
0621 02:41:14.003255 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:14.003298 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:14.004889 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:14.005636 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:14.005644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:14.006048 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:14.006599 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:14.006615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.007555 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.007559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:14.008751 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:14.008761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.009520 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.009525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:14.009711 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:14.010283 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:14.010304 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:14.012208 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:14.015242 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:14.015250 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:14.015264 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:14.015655 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:14.016023 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:14.016028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:14.016032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:14.016229 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:14.016727 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:14.016734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:14.017005 (+ 271us) client_negotiation.cc:769] Sending connection context
0621 02:41:14.017260 (+ 255us) client_negotiation.cc:240] Negotiation successful
0621 02:41:14.017754 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":382,"threads_started":1}
I20260621 02:41:14.019941 13202 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42379
I20260621 02:41:14.020236 13202 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:14.020889 13202 heartbeater.cc:511] Master 127.0.97.62:42379 requested a full tablet report, sending...
I20260621 02:41:14.022220 12712 ts_manager.cc:194] Registered new tserver with Master: 577f571fdb2746b6ad55ad9c476b1ff9 (127.0.97.3:40303)
I20260621 02:41:14.022992 12712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40951
I20260621 02:41:14.027973 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:41:14.029771 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 12792
Jun 21 02:41:14 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:41:14.046046 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:14.030746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.0.97.1:45223)
0621 02:41:14.031382 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:14.031389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:14.031414 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:41:14.031547 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:14.031555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:14.031839 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:14.031960 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:14.033240 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.034287 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.035665 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.036159 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.036253 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:14.040816 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:14.040830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:14.040844 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:14.040882 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:14.043604 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:14.044265 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:14.044272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:14.044274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:14.044410 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:14.044715 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:14.044718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:14.044720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:14.045025 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:14.045254 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:14.045574 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 02:41:14.045770 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":294,"threads_started":1}
I20260621 02:41:14.047403 12920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:14.135193 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 12792
I20260621 02:41:14.151669 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 12931
Jun 21 02:41:14 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:41:14.166492 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:14.152542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.97.2:40621)
0621 02:41:14.153098 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:14.153105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:14.153129 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:41:14.153187 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:14.153193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:14.153455 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:14.153591 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:14.154762 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.155724 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.156947 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.157522 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.157617 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:14.161711 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:14.161721 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:14.161733 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:14.161762 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:14.164264 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:14.164815 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:14.164821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:14.164823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:14.164939 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:14.165236 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:14.165239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:14.165241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:14.165580 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:14.165814 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:14.166034 (+ 220us) server_negotiation.cc:299] Negotiation successful
0621 02:41:14.166219 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":284,"threads_started":1}
I20260621 02:41:14.167629 13063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:14.248816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 12931
I20260621 02:41:14.264348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 13074
Jun 21 02:41:14 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:41:14.280563 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:14.265497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.0.97.3:40303)
0621 02:41:14.266180 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:14.266187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:14.266214 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:41:14.266269 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:14.266275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:14.266600 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:14.266779 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:14.268321 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.269341 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.270528 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:14.271038 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:14.271133 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:14.275082 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:14.275091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:14.275104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:14.275133 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:14.277961 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:14.278609 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:14.278616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:14.278618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:14.278744 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:14.279139 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:14.279142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:14.279144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:14.279521 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:14.279751 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:14.280091 (+ 340us) server_negotiation.cc:299] Negotiation successful
0621 02:41:14.280275 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":378,"threads_started":1}
I20260621 02:41:14.281875 13197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:14.365749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 13074
I20260621 02:41:14.381832 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 12680
I20260621 02:41:14.383121 12742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:14.489521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 12680
I20260621 02:41:14.516637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42379
--webserver_interface=127.0.97.62
--webserver_port=34555
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42379
--ranger_config_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40545/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40545/kms with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:14.829787 13214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:14.830052 13214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:14.830137 13214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:14.836907 13214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:14.837005 13214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:14.837050 13214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:14.837106 13214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:14.837147 13214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:14.846454 13214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40545/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42379
--ranger_config_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:14.847494 13214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:14.848649 13214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:14.860054 13219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:14.860147 13220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:14.860123 13222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:14.860889 13214 server_base.cc:1034] running on GCE node
I20260621 02:41:14.861747 13214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:14.863075 13214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:14.864326 13214 hybrid_clock.cc:648] HybridClock initialized: now 1782009674864295 us; error 47 us; skew 500 ppm
I20260621 02:41:14.864693 13214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:14 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:14.872754 13214 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:41:14.874233 13214 webserver.cc:460] Webserver started at http://127.0.97.62:34555/ using document root <none> and password file <none>
I20260621 02:41:14.874836 13214 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:14.874982 13214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:41:14 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:14.928172 13214 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:41:14.931304 13230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:14.932631 13214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:41:14.932816 13214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "88aeaba99f2e44bbb7047e5f9b275f68"
format_stamp: "Formatted at 2026-06-21 02:41:10 on dist-test-slave-jkn7"
server_key: "d04f060f70edc43193c030074b940ae8"
server_key_iv: "0d54442cd85c6803c4f7d0f12ae3d560"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:14.933810 13214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:14.962499 13214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:14.966768 13214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:14.967080 13214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:14.988720 13214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42379
I20260621 02:41:14.988685 13281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42379 every 8 connection(s)
I20260621 02:41:14.990736 13214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:41:14.998219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 13214
I20260621 02:41:14.999437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45223
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42379
--webserver_port=37873
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40545/kms with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:41:15.000154 13282 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:41:15.005795 13282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Bootstrap starting.
I20260621 02:41:15.028455 13282 log.cc:824] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:15.036852 13282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:41:15.037525 13282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Bootstrap complete.
I20260621 02:41:15.046748 13282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } }
I20260621 02:41:15.047613 13282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88aeaba99f2e44bbb7047e5f9b275f68, State: Initialized, Role: FOLLOWER
I20260621 02:41:15.048257 13282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } }
I20260621 02:41:15.048502 13282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:41:15.048623 13282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:41:15.048815 13282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:41:15.052430 13282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } }
I20260621 02:41:15.052908 13282 leader_election.cc:302] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88aeaba99f2e44bbb7047e5f9b275f68; no voters:
I20260621 02:41:15.053555 13282 leader_election.cc:288] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:41:15.053735 13286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:41:15.055014 13286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [term 2 LEADER]: Becoming Leader. State: Replica: 88aeaba99f2e44bbb7047e5f9b275f68, State: Running, Role: LEADER
I20260621 02:41:15.055541 13286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } }
I20260621 02:41:15.055721 13282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:41:15.057879 13288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88aeaba99f2e44bbb7047e5f9b275f68. Latest consensus state: current_term: 2 leader_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } } }
I20260621 02:41:15.058236 13288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:15.058722 13287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88aeaba99f2e44bbb7047e5f9b275f68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42379 } } }
I20260621 02:41:15.058975 13287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:15.070225 13291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:41:15.076728 13291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:41:15.078624 13291 catalog_manager.cc:1259] Loaded cluster ID: 2d2089637d574c8ea4913229712187a4
I20260621 02:41:15.078792 13291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:41:15.084177 13291 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:41:15.086032 13291 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Loaded TSK: 0
I20260621 02:41:15.087857 13291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:41:15.091082 13282 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260621 02:41:15.437902 13284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:15.438288 13284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:15.438444 13284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:15.450538 13284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:15.450686 13284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:15.450884 13284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:41:15.466596 13284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37873
--tserver_master_addrs=127.0.97.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:15.468246 13284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:15.469978 13284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:15.494865 13320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:15.501784 13321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:15.503981 13324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:15.504480 13284 server_base.cc:1034] running on GCE node
I20260621 02:41:15.505527 13284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:15.510281 13284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:15.511690 13284 hybrid_clock.cc:648] HybridClock initialized: now 1782009675511633 us; error 66 us; skew 500 ppm
I20260621 02:41:15.512146 13284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:15 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:15.516866 13284 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:41:15.533111 13284 webserver.cc:460] Webserver started at http://127.0.97.1:37873/ using document root <none> and password file <none>
I20260621 02:41:15.534411 13284 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:15.534685 13284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:41:15 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:15.592355 13284 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.007s sys 0.008s
I20260621 02:41:15.602602 13331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:15.604398 13284 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260621 02:41:15.604573 13284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "1651ca89d9604dd7b83fc204295b499f"
format_stamp: "Formatted at 2026-06-21 02:41:12 on dist-test-slave-jkn7"
server_key: "7b86ab1f3306fef104ee3464c2ea6044"
server_key_iv: "85c8c2aae4b38ebf58a582cbc53b3220"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:15.606022 13284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:15.660043 13284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:15.665426 13284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:15.665756 13284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:15.667033 13284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:15.669353 13284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:15.669468 13284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:15.669589 13284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:15.669670 13284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:15.714291 13284 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45223
I20260621 02:41:15.714344 13444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45223 every 8 connection(s)
I20260621 02:41:15.715798 13284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:41:15.721537 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 13284
I20260621 02:41:15.722365 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40621
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42379
--webserver_port=34871
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40545/kms with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:41:15 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:15.755205 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:15.725177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42379 (local address 127.0.97.1:58683)
0621 02:41:15.732035 (+ 6858us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:15.732080 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:15.734426 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:15.736281 (+ 1855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:15.736289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:15.736712 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:15.737428 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:15.737447 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:15.739241 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:15.739247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:15.740513 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:15.740524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:15.741981 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:15.741988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:15.742265 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:15.743002 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:15.743039 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:15.745714 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:15.750314 (+ 4600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:15.750324 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:15.750339 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:15.750910 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:15.751356 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:15.751361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:15.751365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:15.751605 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:15.752632 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:15.752640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:15.753006 (+ 366us) client_negotiation.cc:769] Sending connection context
0621 02:41:15.753348 (+ 342us) client_negotiation.cc:240] Negotiation successful
0621 02:41:15.753878 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6557,"thread_start_us":4899,"threads_started":1}
I20260621 02:41:15.757985 13447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:15.726279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58683 (local address 127.0.97.62:42379)
0621 02:41:15.730247 (+ 3968us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:15.730273 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:15.732546 (+ 2273us) server_negotiation.cc:407] Connection header received
0621 02:41:15.735214 (+ 2668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:15.735234 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:15.735739 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:15.736183 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:15.737626 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:15.739071 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:15.740665 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:15.741733 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:15.741943 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:15.746085 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:15.746097 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:15.746116 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:15.746168 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:15.750136 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:15.751074 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:15.751080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:15.751085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:15.751231 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:15.751711 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:15.751716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:15.751719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:15.752357 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:15.752557 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:15.756395 (+ 3838us) server_negotiation.cc:299] Negotiation successful
0621 02:41:15.756767 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3623,"thread_start_us":3333,"threads_started":1}
I20260621 02:41:15.761332 13445 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42379
I20260621 02:41:15.761652 13445 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:15.762331 13445 heartbeater.cc:511] Master 127.0.97.62:42379 requested a full tablet report, sending...
W20260621 02:41:15.764974 13445 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42379 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:41:15.765208 13445 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:15.765487 13445 heartbeater.cc:511] Master 127.0.97.62:42379 requested a full tablet report, sending...
I20260621 02:41:15.766564 13445 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:15.766810 13445 heartbeater.cc:511] Master 127.0.97.62:42379 requested a full tablet report, sending...
W20260621 02:41:15.767784 13445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:41:16.139217 13449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:16.139509 13449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:16.139671 13449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:16.150063 13449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:16.150207 13449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:16.150373 13449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:41:16.160499 13449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34871
--tserver_master_addrs=127.0.97.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:16.161846 13449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:16.163131 13449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:16.178673 13463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:16.179817 13466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:16.180039 13464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:16.179971 13449 server_base.cc:1034] running on GCE node
I20260621 02:41:16.180774 13449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:16.181911 13449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:16.183192 13449 hybrid_clock.cc:648] HybridClock initialized: now 1782009676183124 us; error 47 us; skew 500 ppm
I20260621 02:41:16.183549 13449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:16 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:16.191917 13449 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:41:16.193509 13449 webserver.cc:460] Webserver started at http://127.0.97.2:34871/ using document root <none> and password file <none>
I20260621 02:41:16.194171 13449 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:16.194370 13449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:41:16 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:16.233124 13449 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.007s sys 0.004s
I20260621 02:41:16.248556 13475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:16.250699 13449 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.001s
I20260621 02:41:16.250885 13449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "678b5adb2aae4c68acce1feaa80da6d4"
format_stamp: "Formatted at 2026-06-21 02:41:13 on dist-test-slave-jkn7"
server_key: "3c820b7dd29338c406e9079a71868c76"
server_key_iv: "558faa01178917f5fc69a4a34c50946b"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:16.252288 13449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:16.299532 13449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:16.304478 13449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:16.304767 13449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:16.319615 13449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:16.334270 13449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:16.334414 13449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:16.334534 13449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:16.334610 13449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:41:16 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782009676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:16.481144 13449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40621
I20260621 02:41:16.481207 13589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40621 every 8 connection(s)
I20260621 02:41:16.482946 13449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:41:16.490901 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 13449
I20260621 02:41:16.491685 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40303
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42379
--webserver_port=33547
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40545/kms with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:41:16 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:16.547761 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:16.504388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53813 (local address 127.0.97.62:42379)
0621 02:41:16.514704 (+ 10316us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:16.514714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:16.515848 (+ 1134us) server_negotiation.cc:407] Connection header received
0621 02:41:16.518484 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:16.518492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:16.518583 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:16.518691 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:16.520803 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:16.521976 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:16.533756 (+ 11780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:16.534462 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:16.534594 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:16.540506 (+ 5912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:16.540513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:16.540518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:16.540562 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:16.544119 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:16.545012 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:16.545030 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:16.545034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:16.545179 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:16.545730 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:16.545735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:16.545737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:16.546079 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:16.546201 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:16.547172 (+ 971us) server_negotiation.cc:299] Negotiation successful
0621 02:41:16.547395 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10130,"thread_start_us":10010,"threads_started":1}
I20260621 02:41:16.549891 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:16.503826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42379 (local address 127.0.97.2:53813)
0621 02:41:16.515350 (+ 11524us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:16.515402 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:16.517946 (+ 2544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:16.519106 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:16.519118 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:16.519767 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:16.520577 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:16.520601 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:16.522370 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:16.522377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:16.524731 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:16.524748 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:16.535169 (+ 10421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:16.535178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:16.535519 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:16.537268 (+ 1749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:16.537327 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:16.540217 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:16.544292 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:16.544302 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:16.544320 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:16.544873 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:16.545345 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:16.545349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:16.545353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:16.545603 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:16.546298 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:16.546306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:16.546720 (+ 414us) client_negotiation.cc:769] Sending connection context
0621 02:41:16.547980 (+ 1260us) client_negotiation.cc:240] Negotiation successful
0621 02:41:16.548542 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11219,"thread_start_us":10976,"threads_started":1}
I20260621 02:41:16.551784 13590 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42379
I20260621 02:41:16.552095 13590 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:16.552935 13590 heartbeater.cc:511] Master 127.0.97.62:42379 requested a full tablet report, sending...
I20260621 02:41:16.555061 13247 ts_manager.cc:194] Registered new tserver with Master: 678b5adb2aae4c68acce1feaa80da6d4 (127.0.97.2:40621)
I20260621 02:41:16.557233 13247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53813
I20260621 02:41:16.767946 13445 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:16.768412 13445 heartbeater.cc:511] Master 127.0.97.62:42379 requested a full tablet report, sending...
I20260621 02:41:16.770068 13247 ts_manager.cc:194] Registered new tserver with Master: 1651ca89d9604dd7b83fc204295b499f (127.0.97.1:45223)
I20260621 02:41:16.771533 13247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58683
Jun 21 02:41:16 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260621 02:41:16.938647 13591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:16.939059 13591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:16.939211 13591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:16.950466 13591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:16.950631 13591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:16.950824 13591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:41:16.966181 13591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33547
--tserver_master_addrs=127.0.97.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:16.967911 13591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:16.969599 13591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:17.003566 13600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:17.009740 13599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:17.010316 13603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:17.011370 13591 server_base.cc:1034] running on GCE node
I20260621 02:41:17.012300 13591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:17.021631 13591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:17.025032 13591 hybrid_clock.cc:648] HybridClock initialized: now 1782009677024955 us; error 84 us; skew 500 ppm
I20260621 02:41:17.025480 13591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:17 dist-test-slave-jkn7 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:17.029824 13591 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:41:17.037123 13591 webserver.cc:460] Webserver started at http://127.0.97.3:33547/ using document root <none> and password file <none>
I20260621 02:41:17.038203 13591 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:17.038368 13591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:41:17 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:17.075181 13591 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.010s sys 0.003s
I20260621 02:41:17.089926 13610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:17.098815 13591 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.010s sys 0.005s
I20260621 02:41:17.099134 13591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "577f571fdb2746b6ad55ad9c476b1ff9"
format_stamp: "Formatted at 2026-06-21 02:41:13 on dist-test-slave-jkn7"
server_key: "35cd81b3661dad4fb14bce94ac6585ca"
server_key_iv: "f279128eb5e77ce184212fcc6b252bce"
server_key_version: "kuduclusterkey@0"
I20260621 02:41:17.100827 13591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:17.145109 13591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:17.150049 13591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:17.150391 13591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:17.157745 13591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:17.165845 13591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:17.166010 13591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:17.166174 13591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:17.166281 13591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:41:17 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260621 02:41:17.562394 13590 heartbeater.cc:503] Master 127.0.97.62:42379 was elected leader, sending a full tablet report...
I20260621 02:41:17.632194 13591 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40303
I20260621 02:41:17.632220 13730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40303 every 8 connection(s)
I20260621 02:41:17.633651 13591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:41:17.641098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 13591
Jun 21 02:41:17 dist-test-slave-jkn7 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:17.657833 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.636269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51053 (local address 127.0.97.62:42379)
0621 02:41:17.636786 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:17.636792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:17.637707 (+ 915us) server_negotiation.cc:407] Connection header received
0621 02:41:17.639508 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:17.639514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:17.639620 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:17.639749 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:17.641065 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.642105 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.645237 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.646169 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.646320 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.650383 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:17.650389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:17.650393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:17.650458 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:17.654099 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:17.654996 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:17.655001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:17.655004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:17.655141 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:17.655672 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:17.655678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:17.655685 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:17.655955 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:17.656113 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:17.657174 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0621 02:41:17.657425 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":383,"thread_start_us":240,"threads_started":1}
I20260621 02:41:17.659107 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.636765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42379 (local address 127.0.97.3:51053)
0621 02:41:17.637541 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:17.637586 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:17.639250 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:17.639930 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:17.639938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:17.640346 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:17.640884 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:17.640901 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.642317 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.642326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:17.644896 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:17.644931 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.646734 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.646744 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:17.647126 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.647992 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:17.648015 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:17.650164 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:17.654245 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:17.654255 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:17.654276 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:17.654865 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:17.655271 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:17.655279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:17.655282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:17.655547 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:17.656208 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:17.656217 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:17.656691 (+ 474us) client_negotiation.cc:769] Sending connection context
0621 02:41:17.657076 (+ 385us) client_negotiation.cc:240] Negotiation successful
0621 02:41:17.657636 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":311,"threads_started":1}
I20260621 02:41:17.660679 13731 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42379
I20260621 02:41:17.661015 13731 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:17.661826 13731 heartbeater.cc:511] Master 127.0.97.62:42379 requested a full tablet report, sending...
I20260621 02:41:17.663343 13247 ts_manager.cc:194] Registered new tserver with Master: 577f571fdb2746b6ad55ad9c476b1ff9 (127.0.97.3:40303)
I20260621 02:41:17.664404 13247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51053
I20260621 02:41:17.665817 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.642024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.0.97.62:42379)
0621 02:41:17.650454 (+ 8430us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:17.650458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:17.650471 (+ 13us) server_negotiation.cc:407] Connection header received
0621 02:41:17.650515 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:17.650518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:17.650590 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:17.650716 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:17.652473 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.653492 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.655300 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.655914 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.656030 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.659607 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:17.659614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:17.659619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:17.659662 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:17.662962 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:17.663741 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:17.663746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:17.663749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:17.663914 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:17.664314 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:17.664319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:17.664322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:17.664629 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:17.664867 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:17.665242 (+ 375us) server_negotiation.cc:299] Negotiation successful
0621 02:41:17.665467 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8316,"thread_start_us":7552,"threads_started":1}
I20260621 02:41:17.669642 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:41:17.687088 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.672829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.0.97.62:42379)
0621 02:41:17.673008 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:17.673013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:17.673766 (+ 753us) server_negotiation.cc:407] Connection header received
0621 02:41:17.673906 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:17.673910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:17.673963 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:17.674042 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:17.675842 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.676501 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.678392 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.678809 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.678909 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.681958 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:17.681963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:17.681966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:17.682014 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:17.684589 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:17.685384 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:17.685388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:17.685390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:17.685502 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:17.685890 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:17.685893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:17.685895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:17.686126 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:17.686295 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:17.686646 (+ 351us) server_negotiation.cc:299] Negotiation successful
0621 02:41:17.686803 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:41:17.691982 13247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:41:17.699461 13247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:41:17.747258 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.714100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40621 (local address 127.0.0.1:33812)
0621 02:41:17.732292 (+ 18192us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:17.732361 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:17.732597 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:17.739227 (+ 6630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:17.739231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:17.739273 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:17.739768 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:17.739783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.742156 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.742162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:17.744354 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:17.744368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.746191 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.746198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:17.746395 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.746453 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 02:41:17.746570 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 02:41:17.746785 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17801,"mutex_wait_us":85,"thread_start_us":17625,"threads_started":1}
I20260621 02:41:17.747893 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.715711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.0.97.1:45223)
0621 02:41:17.737383 (+ 21672us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:17.737396 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:17.737430 (+ 34us) server_negotiation.cc:407] Connection header received
0621 02:41:17.737548 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:17.737557 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:17.737868 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:17.738742 (+ 874us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:17.739930 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.741518 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.744410 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.746138 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.746260 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.746507 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:17.747358 (+ 851us) server_negotiation.cc:299] Negotiation successful
0621 02:41:17.747531 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21456,"thread_start_us":21261,"threads_started":1}
I20260621 02:41:17.748565 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.715328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45223 (local address 127.0.0.1:48016)
0621 02:41:17.734982 (+ 19654us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:17.735006 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:17.735163 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:17.739017 (+ 3854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:17.739025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:17.739094 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:17.739707 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:17.739726 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.741681 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.741686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:17.744265 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:17.744282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.746668 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.746675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:17.746859 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.746893 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:41:17.746983 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 02:41:17.747124 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19491,"mutex_wait_us":95,"spinlock_wait_cycles":3456,"thread_start_us":19339,"threads_started":1}
I20260621 02:41:17.748668 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.719848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40303 (local address 127.0.0.1:44092)
0621 02:41:17.738300 (+ 18452us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:17.738319 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:17.738443 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:17.739039 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:17.739043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:17.739106 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:17.740045 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:17.740053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.742069 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.742075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:17.746070 (+ 3995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:17.746087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.747977 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.747986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:17.748177 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.748211 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:41:17.748263 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 02:41:17.748343 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18337,"mutex_wait_us":1749,"spinlock_wait_cycles":23296,"thread_start_us":18147,"threads_started":1}
I20260621 02:41:17.749722 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.720617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.0.97.3:40303)
0621 02:41:17.737835 (+ 17218us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:17.737843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:17.738436 (+ 593us) server_negotiation.cc:407] Connection header received
0621 02:41:17.738578 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:17.738585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:17.738817 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:17.739598 (+ 781us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:17.740199 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.741893 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.746290 (+ 4397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.747737 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.748996 (+ 1259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.749162 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:17.749211 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 02:41:17.749395 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17049,"thread_start_us":16979,"threads_started":1}
I20260621 02:41:17.750293 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:17.714612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.0.97.2:40621)
0621 02:41:17.738502 (+ 23890us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:17.738512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:17.738535 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:41:17.738616 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:17.738622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:17.738814 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:17.740257 (+ 1443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:17.740497 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.742002 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.744483 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:17.745922 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:17.749509 (+ 3587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:17.749738 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:17.749816 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 02:41:17.749980 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23698,"thread_start_us":24343,"threads_started":1}
I20260621 02:41:17.752651 13380 tablet_service.cc:1467] Processing CreateTablet for tablet 825f3dd50c884a688d6d5885aedc6738 (DEFAULT_TABLE table=test-table [id=eec8a2267d70496d8139b7a73a8b0366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:17.753624 13524 tablet_service.cc:1467] Processing CreateTablet for tablet 825f3dd50c884a688d6d5885aedc6738 (DEFAULT_TABLE table=test-table [id=eec8a2267d70496d8139b7a73a8b0366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:17.753624 13659 tablet_service.cc:1467] Processing CreateTablet for tablet 825f3dd50c884a688d6d5885aedc6738 (DEFAULT_TABLE table=test-table [id=eec8a2267d70496d8139b7a73a8b0366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:17.754253 13380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825f3dd50c884a688d6d5885aedc6738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:17.754890 13524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825f3dd50c884a688d6d5885aedc6738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:17.754887 13659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825f3dd50c884a688d6d5885aedc6738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:17.774844 13753 tablet_bootstrap.cc:492] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: Bootstrap starting.
I20260621 02:41:17.775661 13445 heartbeater.cc:503] Master 127.0.97.62:42379 was elected leader, sending a full tablet report...
I20260621 02:41:17.779439 13753 tablet_bootstrap.cc:654] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:17.780828 13753 log.cc:824] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:17.781917 13752 tablet_bootstrap.cc:492] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: Bootstrap starting.
I20260621 02:41:17.786371 13752 tablet_bootstrap.cc:654] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:17.787354 13754 tablet_bootstrap.cc:492] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: Bootstrap starting.
I20260621 02:41:17.788033 13752 log.cc:824] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:17.790675 13752 tablet_bootstrap.cc:492] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: No bootstrap required, opened a new log
I20260621 02:41:17.790668 13754 tablet_bootstrap.cc:654] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:17.790937 13752 ts_tablet_manager.cc:1397] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 02:41:17.791805 13754 log.cc:824] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:17.794641 13754 tablet_bootstrap.cc:492] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: No bootstrap required, opened a new log
I20260621 02:41:17.794780 13753 tablet_bootstrap.cc:492] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: No bootstrap required, opened a new log
I20260621 02:41:17.794950 13754 ts_tablet_manager.cc:1397] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 02:41:17.795116 13753 ts_tablet_manager.cc:1397] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.004s
I20260621 02:41:17.800936 13752 raft_consensus.cc:348] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } }
I20260621 02:41:17.801509 13752 raft_consensus.cc:374] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:17.801645 13752 raft_consensus.cc:729] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1651ca89d9604dd7b83fc204295b499f, State: Initialized, Role: FOLLOWER
I20260621 02:41:17.802398 13752 consensus_queue.cc:260] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } }
I20260621 02:41:17.804163 13752 ts_tablet_manager.cc:1428] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260621 02:41:17.805392 13754 raft_consensus.cc:348] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } }
I20260621 02:41:17.805827 13754 raft_consensus.cc:374] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:17.805948 13754 raft_consensus.cc:729] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678b5adb2aae4c68acce1feaa80da6d4, State: Initialized, Role: FOLLOWER
I20260621 02:41:17.806663 13754 consensus_queue.cc:260] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } }
I20260621 02:41:17.807008 13753 raft_consensus.cc:348] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } }
I20260621 02:41:17.807440 13753 raft_consensus.cc:374] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:17.807556 13753 raft_consensus.cc:729] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577f571fdb2746b6ad55ad9c476b1ff9, State: Initialized, Role: FOLLOWER
I20260621 02:41:17.808137 13753 consensus_queue.cc:260] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } }
I20260621 02:41:17.809183 13731 heartbeater.cc:503] Master 127.0.97.62:42379 was elected leader, sending a full tablet report...
I20260621 02:41:17.809860 13753 ts_tablet_manager.cc:1428] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260621 02:41:17.810411 13754 ts_tablet_manager.cc:1428] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260621 02:41:17.885365 13732 tablet.cc:2369] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:41:17.986968 13446 tablet.cc:2369] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:41:18.035604 13592 tablet.cc:2369] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:41:18.036273 13759 raft_consensus.cc:482] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:18.036599 13759 raft_consensus.cc:504] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } }
I20260621 02:41:18.038465 13759 leader_election.cc:288] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1651ca89d9604dd7b83fc204295b499f (127.0.97.1:45223), 577f571fdb2746b6ad55ad9c476b1ff9 (127.0.97.3:40303)
I20260621 02:41:18.045382 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:18.039073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34199 (local address 127.0.97.3:40303)
0621 02:41:18.039327 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:18.039337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:18.039964 (+ 627us) server_negotiation.cc:407] Connection header received
0621 02:41:18.040146 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:18.040150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:18.040207 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:18.040321 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:18.040957 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.041719 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.043553 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.044291 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.044385 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:18.044459 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:18.044964 (+ 505us) server_negotiation.cc:299] Negotiation successful
0621 02:41:18.045095 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:41:18.045456 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:18.038812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41461 (local address 127.0.97.1:45223)
0621 02:41:18.039029 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:18.039037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:18.039798 (+ 761us) server_negotiation.cc:407] Connection header received
0621 02:41:18.040070 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:18.040077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:18.040172 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:18.040242 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:18.040896 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.041690 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.043589 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.044473 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.044613 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:18.044710 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:18.045055 (+ 345us) server_negotiation.cc:299] Negotiation successful
0621 02:41:18.045204 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:41:18.045522 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:18.038948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40303 (local address 127.0.97.2:34199)
0621 02:41:18.039854 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:18.039872 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:18.040020 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:18.040355 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:18.040361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:18.040391 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:18.040775 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:18.040783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.041883 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.041886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:18.043387 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:18.043402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.044597 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.044603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:18.044808 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:18.044832 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 02:41:18.044906 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 02:41:18.045032 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":1,"thread_start_us":602,"threads_started":1}
I20260621 02:41:18.045939 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:18.038826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45223 (local address 127.0.97.2:41461)
0621 02:41:18.039667 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:18.039686 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:18.039868 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:18.040289 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:18.040296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:18.040325 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:18.040738 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:18.040749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.041815 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.041824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:18.043413 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:18.043424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.044683 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.044687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:18.044860 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:18.044904 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 02:41:18.045000 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 02:41:18.045133 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":115,"thread_start_us":487,"threads_started":1}
I20260621 02:41:18.046653 13679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825f3dd50c884a688d6d5885aedc6738" candidate_uuid: "678b5adb2aae4c68acce1feaa80da6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" is_pre_election: true
I20260621 02:41:18.046911 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825f3dd50c884a688d6d5885aedc6738" candidate_uuid: "678b5adb2aae4c68acce1feaa80da6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1651ca89d9604dd7b83fc204295b499f" is_pre_election: true
I20260621 02:41:18.047263 13679 raft_consensus.cc:2454] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 678b5adb2aae4c68acce1feaa80da6d4 in term 0.
I20260621 02:41:18.047434 13400 raft_consensus.cc:2454] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 678b5adb2aae4c68acce1feaa80da6d4 in term 0.
I20260621 02:41:18.048079 13479 leader_election.cc:302] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 577f571fdb2746b6ad55ad9c476b1ff9, 678b5adb2aae4c68acce1feaa80da6d4; no voters:
I20260621 02:41:18.048441 13759 raft_consensus.cc:2781] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:41:18.048588 13759 raft_consensus.cc:482] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:41:18.048687 13759 raft_consensus.cc:3037] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:18.050887 13759 raft_consensus.cc:504] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } }
I20260621 02:41:18.051455 13759 leader_election.cc:288] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [CANDIDATE]: Term 1 election: Requested vote from peers 1651ca89d9604dd7b83fc204295b499f (127.0.97.1:45223), 577f571fdb2746b6ad55ad9c476b1ff9 (127.0.97.3:40303)
I20260621 02:41:18.051865 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825f3dd50c884a688d6d5885aedc6738" candidate_uuid: "678b5adb2aae4c68acce1feaa80da6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1651ca89d9604dd7b83fc204295b499f"
I20260621 02:41:18.051954 13679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825f3dd50c884a688d6d5885aedc6738" candidate_uuid: "678b5adb2aae4c68acce1feaa80da6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "577f571fdb2746b6ad55ad9c476b1ff9"
I20260621 02:41:18.052058 13400 raft_consensus.cc:3037] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:18.052192 13679 raft_consensus.cc:3037] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:18.054235 13400 raft_consensus.cc:2454] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678b5adb2aae4c68acce1feaa80da6d4 in term 1.
I20260621 02:41:18.054467 13679 raft_consensus.cc:2454] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678b5adb2aae4c68acce1feaa80da6d4 in term 1.
I20260621 02:41:18.054721 13480 leader_election.cc:302] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1651ca89d9604dd7b83fc204295b499f, 678b5adb2aae4c68acce1feaa80da6d4; no voters:
I20260621 02:41:18.055029 13759 raft_consensus.cc:2781] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:41:18.055487 13759 raft_consensus.cc:686] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 678b5adb2aae4c68acce1feaa80da6d4, State: Running, Role: LEADER
I20260621 02:41:18.055956 13759 consensus_queue.cc:237] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } }
I20260621 02:41:18.060446 13247 catalog_manager.cc:5495] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 678b5adb2aae4c68acce1feaa80da6d4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "678b5adb2aae4c68acce1feaa80da6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b5adb2aae4c68acce1feaa80da6d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 } health_report { overall_health: UNKNOWN } } }
I20260621 02:41:18.095566 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:18.090303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.0.97.2:40621)
0621 02:41:18.090552 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:18.090558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:18.090577 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:41:18.090715 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:18.090719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:18.090794 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:18.090941 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:41:18.091313 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.092036 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.093465 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.093885 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.094010 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:18.094526 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:41:18.095000 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:41:18.095164 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:18.095220 (+ 56us) server_negotiation.cc:299] Negotiation successful
0621 02:41:18.095298 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260621 02:41:18.103430 13679 raft_consensus.cc:1261] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [term 1 FOLLOWER]: Refusing update from remote peer 678b5adb2aae4c68acce1feaa80da6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:18.103430 13400 raft_consensus.cc:1261] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [term 1 FOLLOWER]: Refusing update from remote peer 678b5adb2aae4c68acce1feaa80da6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:18.104357 13759 consensus_queue.cc:1035] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "577f571fdb2746b6ad55ad9c476b1ff9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:41:18.104712 13763 consensus_queue.cc:1035] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1651ca89d9604dd7b83fc204295b499f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:41:18.115589 13766 mvcc.cc:205] Tried to move back new op lower bound from 7299111641502416896 to 7299111641320902656. Current Snapshot: MvccSnapshot[applied={T|T < 7299111641502416896}]
I20260621 02:41:18.127862 13246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:41:18.128161 13246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:41:18.131572 13246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88aeaba99f2e44bbb7047e5f9b275f68: Sending DeleteTablet for 3 replicas of tablet 825f3dd50c884a688d6d5885aedc6738
I20260621 02:41:18.132609 13524 tablet_service.cc:1514] Processing DeleteTablet for tablet 825f3dd50c884a688d6d5885aedc6738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33812
I20260621 02:41:18.132640 13380 tablet_service.cc:1514] Processing DeleteTablet for tablet 825f3dd50c884a688d6d5885aedc6738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48016
I20260621 02:41:18.132911 13659 tablet_service.cc:1514] Processing DeleteTablet for tablet 825f3dd50c884a688d6d5885aedc6738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44092
I20260621 02:41:18.138180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 13284
I20260621 02:41:18.148363 13777 tablet_replica.cc:330] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: stopping tablet replica
I20260621 02:41:18.148612 13776 tablet_replica.cc:330] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: stopping tablet replica
I20260621 02:41:18.148813 13777 raft_consensus.cc:2229] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:41:18.148990 13776 raft_consensus.cc:2229] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:41:18.149467 13776 raft_consensus.cc:2258] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:18.149956 13777 raft_consensus.cc:2258] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:18.150192 13775 tablet_replica.cc:330] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: stopping tablet replica
I20260621 02:41:18.150548 13775 raft_consensus.cc:2229] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:41:18.151201 13775 raft_consensus.cc:2258] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:18.153255 13776 ts_tablet_manager.cc:1905] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:18.153509 13775 ts_tablet_manager.cc:1905] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:18.155818 13777 ts_tablet_manager.cc:1905] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:18.159719 13775 ts_tablet_manager.cc:1918] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:18.160181 13775 log.cc:1196] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/825f3dd50c884a688d6d5885aedc6738
I20260621 02:41:18.160764 13775 ts_tablet_manager.cc:1939] T 825f3dd50c884a688d6d5885aedc6738 P 678b5adb2aae4c68acce1feaa80da6d4: Deleting consensus metadata
I20260621 02:41:18.162408 13776 ts_tablet_manager.cc:1918] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:18.162528 13231 catalog_manager.cc:4841] TS 678b5adb2aae4c68acce1feaa80da6d4 (127.0.97.2:40621): tablet 825f3dd50c884a688d6d5885aedc6738 (table test-table [id=eec8a2267d70496d8139b7a73a8b0366]) successfully deleted
I20260621 02:41:18.162642 13777 ts_tablet_manager.cc:1918] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:18.162635 13776 log.cc:1196] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/825f3dd50c884a688d6d5885aedc6738
I20260621 02:41:18.162850 13777 log.cc:1196] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/825f3dd50c884a688d6d5885aedc6738
I20260621 02:41:18.163297 13776 ts_tablet_manager.cc:1939] T 825f3dd50c884a688d6d5885aedc6738 P 1651ca89d9604dd7b83fc204295b499f: Deleting consensus metadata
I20260621 02:41:18.163388 13777 ts_tablet_manager.cc:1939] T 825f3dd50c884a688d6d5885aedc6738 P 577f571fdb2746b6ad55ad9c476b1ff9: Deleting consensus metadata
I20260621 02:41:18.164934 13234 catalog_manager.cc:4841] TS 1651ca89d9604dd7b83fc204295b499f (127.0.97.1:45223): tablet 825f3dd50c884a688d6d5885aedc6738 (table test-table [id=eec8a2267d70496d8139b7a73a8b0366]) successfully deleted
I20260621 02:41:18.164991 13233 catalog_manager.cc:4841] TS 577f571fdb2746b6ad55ad9c476b1ff9 (127.0.97.3:40303): tablet 825f3dd50c884a688d6d5885aedc6738 (table test-table [id=eec8a2267d70496d8139b7a73a8b0366]) successfully deleted
I20260621 02:41:18.166206 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:18.139093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.0.97.1:45223)
0621 02:41:18.139288 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:18.139297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:18.139684 (+ 387us) server_negotiation.cc:407] Connection header received
0621 02:41:18.139898 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:18.139906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:18.140042 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:18.140174 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:18.147192 (+ 7018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.148505 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.150449 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.151060 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.151196 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:18.154583 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:18.154596 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:18.154613 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:18.154648 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:18.158377 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:18.163631 (+ 5254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:18.163643 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:18.163649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:18.163819 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:18.164268 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:18.164273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:18.164276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:18.164710 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:18.165007 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:18.165612 (+ 605us) server_negotiation.cc:299] Negotiation successful
0621 02:41:18.165827 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 02:41:18.166847 13440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:18.254436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 13284
I20260621 02:41:18.271601 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 13449
I20260621 02:41:18.291496 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:18.272516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.0.97.2:40621)
0621 02:41:18.272698 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:18.272704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:18.272725 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:41:18.272814 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:18.272819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:18.272898 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:18.273006 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:18.274264 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.275511 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.276806 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.277685 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.277809 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:18.284604 (+ 6795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:18.284621 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:18.284642 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:18.284686 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:18.288514 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:18.289155 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:18.289163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:18.289167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:18.289378 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:18.289841 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:18.289846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:18.289849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:18.290306 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:18.290611 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:18.290883 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 02:41:18.291118 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:41:18.292327 13585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:18.404708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 13449
I20260621 02:41:18.422879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 13591
I20260621 02:41:18.435770 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:18.423718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.0.97.3:40303)
0621 02:41:18.423898 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:18.423905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:18.423926 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:41:18.424002 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:18.424006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:18.424068 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:18.424184 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:18.425484 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.426194 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.427446 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:18.427836 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:18.427962 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:18.430427 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:18.430440 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:18.430455 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:18.430492 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:18.433043 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:18.433795 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:18.433812 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:18.433814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:18.433942 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:18.434316 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:18.434320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:18.434322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:18.434660 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:18.434965 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:18.435239 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 02:41:18.435443 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:41:18.436386 13726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:18.523366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 13591
I20260621 02:41:18.540275 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 13214
I20260621 02:41:18.541512 13277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:18.650869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 13214
I20260621 02:41:18.672890 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 02:41:19.007634 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 02:41:19.007824 388 mini_ranger.cc:66] Stopping Ranger...
I20260621 02:41:19.352612 388 mini_ranger.cc:68] Stopped Ranger
2026-06-21 02:41:19.352 UTC [12266] LOG: received smart shutdown request
2026-06-21 02:41:19.355 UTC [12266] LOG: background worker "logical replication launcher" (PID 12275) exited with exit code 1
2026-06-21 02:41:19.356 UTC [12270] LOG: shutting down
2026-06-21 02:41:19.508 UTC [12266] LOG: database system is shut down
2026-06-21T02:41:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51780 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:41:19 dist-test-slave-jkn7 krb5kdc[13790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:41:19 dist-test-slave-jkn7 krb5kdc[13790](info): set up 2 sockets
Jun 21 02:41:19 dist-test-slave-jkn7 krb5kdc[13790](info): commencing operation
krb5kdc: starting...
W20260621 02:41:20.729779 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:41:20 dist-test-slave-jkn7 krb5kdc[13790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:41:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:41:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:41:20.953034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35155
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35155,127.0.97.61:37087,127.0.97.60:39289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:21.279007 13803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:21.279363 13803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:21.279533 13803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:21.286924 13803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:21.287082 13803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:21.287199 13803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:21.287298 13803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:21.287385 13803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:21.297205 13803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35155,127.0.97.61:37087,127.0.97.60:39289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:21.298565 13803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:21.299893 13803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:21.311975 13808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:21.311945 13809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:21.311852 13811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:21.313342 13803 server_base.cc:1034] running on GCE node
I20260621 02:41:21.314284 13803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:21.316005 13803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:21.317267 13803 hybrid_clock.cc:648] HybridClock initialized: now 1782009681317210 us; error 43 us; skew 500 ppm
I20260621 02:41:21.317643 13803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:21 dist-test-slave-jkn7 krb5kdc[13790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:21.324746 13803 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:41:21.326296 13803 webserver.cc:460] Webserver started at http://127.0.97.62:46443/ using document root <none> and password file <none>
I20260621 02:41:21.326974 13803 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:21.327126 13803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:21.327421 13803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:21.330418 13803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "eee77737809b4a6b8c7d1ba0ba4eab21"
format_stamp: "Formatted at 2026-06-21 02:41:21 on dist-test-slave-jkn7"
server_key: "ae31758b98b593ef5304998b65a07e7f"
server_key_iv: "aab4eaf3fecf272d27053e70a6131812"
server_key_version: "encryptionkey@0"
I20260621 02:41:21.331190 13803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "eee77737809b4a6b8c7d1ba0ba4eab21"
format_stamp: "Formatted at 2026-06-21 02:41:21 on dist-test-slave-jkn7"
server_key: "ae31758b98b593ef5304998b65a07e7f"
server_key_iv: "aab4eaf3fecf272d27053e70a6131812"
server_key_version: "encryptionkey@0"
I20260621 02:41:21.336254 13803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:41:21.339669 13818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:21.340968 13803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260621 02:41:21.341166 13803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "eee77737809b4a6b8c7d1ba0ba4eab21"
format_stamp: "Formatted at 2026-06-21 02:41:21 on dist-test-slave-jkn7"
server_key: "ae31758b98b593ef5304998b65a07e7f"
server_key_iv: "aab4eaf3fecf272d27053e70a6131812"
server_key_version: "encryptionkey@0"
I20260621 02:41:21.341387 13803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:21.371495 13803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:21.375752 13803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:21.376047 13803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:21.394469 13803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35155
I20260621 02:41:21.394508 13869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35155 every 8 connection(s)
I20260621 02:41:21.395862 13803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:41:21.399348 13870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:21.405589 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 13803
I20260621 02:41:21.405795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:41:21.405543 13870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:41:21.406452 388 external_mini_cluster.cc:1428] Setting key 841b5fa1b29fb9c5792eb3a14f8a5455
Jun 21 02:41:21 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260621 02:41:21.427469 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.410023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.0.97.62:35155)
0621 02:41:21.410783 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:21.410791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:21.410848 (+ 57us) server_negotiation.cc:407] Connection header received
0621 02:41:21.411938 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:21.411944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:21.412143 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:21.412357 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:21.413630 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.414823 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.416204 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.416754 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.416954 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:21.420705 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:21.420713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:21.420724 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:21.420755 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:21.423493 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:21.424350 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:21.424356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:21.424358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:21.424494 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:21.424933 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:21.424936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:21.424937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:21.425474 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:21.425629 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:21.426274 (+ 645us) server_negotiation.cc:299] Negotiation successful
0621 02:41:21.426500 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":374,"threads_started":1}
I20260621 02:41:21.427588 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.408754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35155 (local address 127.0.0.1:42014)
0621 02:41:21.409561 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.409604 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:21.411574 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:21.412359 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:21.412365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:21.413055 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:21.413441 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:21.413460 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.414979 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.414985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:21.416060 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:21.416076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.416988 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.416995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:21.417183 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:21.417876 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:21.417902 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:21.420482 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:21.423634 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:21.423647 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:21.423658 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:21.424220 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:21.424616 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:21.424623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:21.424628 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:21.424816 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:21.425623 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:21.425630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:21.425859 (+ 229us) client_negotiation.cc:769] Sending connection context
0621 02:41:21.426144 (+ 285us) client_negotiation.cc:240] Negotiation successful
0621 02:41:21.426628 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":118,"thread_start_us":340,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260621 02:41:21.430608 13870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:41:21.431703 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.431326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:50654)
0621 02:41:21.431491 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.431601 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260621 02:41:21.432001 13819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
I20260621 02:41:21.433580 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.433215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:50668)
0621 02:41:21.433374 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.433459 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:41:21.433795 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37087: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260621 02:41:21.452982 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37087
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35155,127.0.97.61:37087,127.0.97.60:39289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:41:21.483120 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } attempt: 1
I20260621 02:41:21.484293 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.483851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35330)
0621 02:41:21.484045 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.484157 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 02:41:21.485378 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.484925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35336)
0621 02:41:21.485079 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.485147 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 02:41:21.485565 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37087: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
I20260621 02:41:21.553834 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } attempt: 2
I20260621 02:41:21.555001 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.554552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35340)
0621 02:41:21.554755 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.554856 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:41:21.556052 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.555688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35346)
0621 02:41:21.555832 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.555922 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:41:21.556316 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37087: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
I20260621 02:41:21.647609 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } attempt: 3
I20260621 02:41:21.648991 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.648439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35362)
0621 02:41:21.648744 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.648843 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260621 02:41:21.650084 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.649705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35378)
0621 02:41:21.649860 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.649941 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 02:41:21.650285 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37087: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
W20260621 02:41:21.773708 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:21.774066 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:21.774222 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:21.781440 13877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:21.781571 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:21.781677 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:21.781780 13877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:21.781862 13877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:21.791455 13877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35155,127.0.97.61:37087,127.0.97.60:39289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:21.792737 13877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:21.793529 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } attempt: 4
I20260621 02:41:21.794097 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:41:21.794729 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.794155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35388)
0621 02:41:21.794510 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.794597 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260621 02:41:21.795713 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.795354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35396)
0621 02:41:21.795497 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.795591 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:41:21.795899 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37087: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37087: connect: Connection refused (error 111)
W20260621 02:41:21.806416 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:21.806550 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:21.806521 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:21.807426 13877 server_base.cc:1034] running on GCE node
I20260621 02:41:21.808197 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:21.809654 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:21.810909 13877 hybrid_clock.cc:648] HybridClock initialized: now 1782009681810822 us; error 79 us; skew 500 ppm
I20260621 02:41:21.811293 13877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:21 dist-test-slave-jkn7 krb5kdc[13790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:21.814379 13877 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260621 02:41:21.815850 13877 webserver.cc:460] Webserver started at http://127.0.97.61:38491/ using document root <none> and password file <none>
I20260621 02:41:21.816525 13877 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:21.816673 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:21.816977 13877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:21.819923 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/data/instance:
uuid: "b65939eb8af8443aa6976b9fad779bab"
format_stamp: "Formatted at 2026-06-21 02:41:21 on dist-test-slave-jkn7"
server_key: "bed26a6f3098f8e6234978d50eb4fb45"
server_key_iv: "1246dc7b6fd3b60b66948042972e0ae4"
server_key_version: "encryptionkey@0"
I20260621 02:41:21.820706 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/wal/instance:
uuid: "b65939eb8af8443aa6976b9fad779bab"
format_stamp: "Formatted at 2026-06-21 02:41:21 on dist-test-slave-jkn7"
server_key: "bed26a6f3098f8e6234978d50eb4fb45"
server_key_iv: "1246dc7b6fd3b60b66948042972e0ae4"
server_key_version: "encryptionkey@0"
I20260621 02:41:21.825753 13877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:41:21.829097 13892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:21.830458 13877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:41:21.830652 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/wal
uuid: "b65939eb8af8443aa6976b9fad779bab"
format_stamp: "Formatted at 2026-06-21 02:41:21 on dist-test-slave-jkn7"
server_key: "bed26a6f3098f8e6234978d50eb4fb45"
server_key_iv: "1246dc7b6fd3b60b66948042972e0ae4"
server_key_version: "encryptionkey@0"
I20260621 02:41:21.830852 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:21.863651 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:21.867529 13877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:21.867826 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:21.885864 13877 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37087
I20260621 02:41:21.885910 13943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37087 every 8 connection(s)
I20260621 02:41:21.887187 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
I20260621 02:41:21.891324 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:21.893046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 13877
I20260621 02:41:21.893253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-1/wal/instance
I20260621 02:41:21.893720 388 external_mini_cluster.cc:1428] Setting key 94f840451ab2d2cc096352ff249ed16f
I20260621 02:41:21.897931 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:41:21 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260621 02:41:21.918695 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.900540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.0.97.62:35155)
0621 02:41:21.900751 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:21.900758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:21.901992 (+ 1234us) server_negotiation.cc:407] Connection header received
0621 02:41:21.903829 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:21.903834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:21.903929 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:21.904064 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:21.905532 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.906223 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.908030 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.908464 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.908604 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:21.912620 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:21.912626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:21.912630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:21.912665 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:21.915628 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:21.916360 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:21.916366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:21.916370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:21.916494 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:21.916935 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:21.916939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:21.916944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:21.917218 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:21.917419 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:21.918125 (+ 706us) server_negotiation.cc:299] Negotiation successful
0621 02:41:21.918364 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:41:21.919613 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.901054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35155 (local address 127.0.0.1:35054)
0621 02:41:21.901837 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.901881 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:21.903562 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:21.904276 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:21.904284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:21.904773 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:21.905371 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:21.905389 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.906378 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.906390 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:21.907890 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:21.907901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.908696 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.908703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:21.908939 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:21.909644 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:21.909681 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:21.912380 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:21.915787 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:21.915795 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:21.915811 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:21.916224 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:21.916609 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:21.916614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:21.916617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:21.916807 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:21.917394 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:21.917401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:21.917746 (+ 345us) client_negotiation.cc:769] Sending connection context
0621 02:41:21.918017 (+ 271us) client_negotiation.cc:240] Negotiation successful
0621 02:41:21.918517 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":364,"threads_started":1}
I20260621 02:41:21.921509 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:41:21 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260621 02:41:21.936635 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.922345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35398)
0621 02:41:21.922519 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.922538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:21.922635 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:21.923663 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:21.923667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:21.924088 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:21.924211 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:21.924218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.925606 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.925613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:21.926415 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:21.926425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.927234 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.927240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:21.927453 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:21.928062 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:21.928080 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:21.930366 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:21.933928 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:21.933937 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:21.933941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:21.934298 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:21.934819 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:21.934824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:21.934827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:21.934998 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:21.935652 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:21.935657 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:21.935797 (+ 140us) client_negotiation.cc:769] Sending connection context
0621 02:41:21.935951 (+ 154us) client_negotiation.cc:240] Negotiation successful
0621 02:41:21.936232 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":3}
I20260621 02:41:21.936797 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.922595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.97.61:37087)
0621 02:41:21.923274 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:21.923280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:21.923304 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:41:21.923344 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:21.923349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:21.923510 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:21.923672 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:21.924363 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.925460 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.926562 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:21.927084 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:21.927225 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:21.930591 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:21.930601 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:21.930614 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:21.930645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:21.933808 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:21.934493 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:21.934502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:21.934505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:21.934653 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:21.935103 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:21.935119 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:21.935128 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:21.935530 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:21.935772 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:21.936096 (+ 324us) server_negotiation.cc:299] Negotiation successful
0621 02:41:21.936390 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":544,"thread_start_us":403,"threads_started":1}
I20260621 02:41:21.939085 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:41:21.940233 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.939769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60504)
0621 02:41:21.939994 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.940097 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260621 02:41:21.940491 13894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260621 02:41:21.942091 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.941711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60510)
0621 02:41:21.941865 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.941964 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:41:21.942301 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
I20260621 02:41:21.942776 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39289
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35155,127.0.97.61:37087,127.0.97.60:39289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:41:21.991660 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } attempt: 1
I20260621 02:41:21.992861 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.992413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60516)
0621 02:41:21.992610 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.992712 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 02:41:21.993993 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:21.993681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60532)
0621 02:41:21.993826 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:21.993890 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:41:21.994243 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
I20260621 02:41:22.062597 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } attempt: 2
I20260621 02:41:22.063819 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.063334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60534)
0621 02:41:22.063570 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.063661 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260621 02:41:22.064929 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.064616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60548)
0621 02:41:22.064752 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.064820 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:41:22.065186 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
I20260621 02:41:22.095198 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } attempt: 5
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260621 02:41:22.110196 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.095966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35410)
0621 02:41:22.096230 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.096257 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:22.096425 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:22.096842 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:22.096849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:22.097429 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:22.097670 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.097684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.098664 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.098669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.100030 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.100045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.100788 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.100795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.100994 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.101670 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:22.101693 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:22.104241 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:22.107508 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.107514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.107518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.108090 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.108481 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.108487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.108491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.108644 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.109177 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:22.109181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:22.109376 (+ 195us) client_negotiation.cc:769] Sending connection context
0621 02:41:22.109537 (+ 161us) client_negotiation.cc:240] Negotiation successful
0621 02:41:22.109834 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 02:41:22.110198 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.096118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.0.97.61:37087)
0621 02:41:22.096272 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.096276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.096347 (+ 71us) server_negotiation.cc:407] Connection header received
0621 02:41:22.096587 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.096591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.096654 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.096737 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.097815 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.098511 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.100186 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.100629 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.100729 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.104429 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.104434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.104437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.104469 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.107361 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.108233 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.108238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.108240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.108359 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.108755 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.108762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.108765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.109056 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.109215 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.109658 (+ 443us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.109872 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 02:41:22.111375 13870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:41:22.112699 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.112266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60550)
0621 02:41:22.112457 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.112544 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 02:41:22.113749 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.113434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60558)
0621 02:41:22.113573 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.113649 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 02:41:22.114028 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
I20260621 02:41:22.156414 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } attempt: 3
I20260621 02:41:22.157663 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.157172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60560)
0621 02:41:22.157411 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.157498 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 02:41:22.158694 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.158302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60562)
0621 02:41:22.158507 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.158586 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260621 02:41:22.158947 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
I20260621 02:41:22.165266 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } attempt: 1
I20260621 02:41:22.166290 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.165892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60578)
0621 02:41:22.166066 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.166150 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 02:41:22.167198 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.166868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60590)
0621 02:41:22.167025 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.167102 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:41:22.167399 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
I20260621 02:41:22.235652 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } attempt: 2
I20260621 02:41:22.236809 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.236331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60606)
0621 02:41:22.236547 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.236647 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 02:41:22.237903 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.237591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60608)
0621 02:41:22.237723 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.237791 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:41:22.238210 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
W20260621 02:41:22.268488 13951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:22.268752 13951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:22.268904 13951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:22.276154 13951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:22.276275 13951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:22.276374 13951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:22.276453 13951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:22.276520 13951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:22.286062 13951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35155,127.0.97.61:37087,127.0.97.60:39289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:22.287323 13951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:22.288578 13951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:22.300112 13956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:22.300170 13959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:22.300202 13957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:22.300921 13951 server_base.cc:1034] running on GCE node
I20260621 02:41:22.301782 13951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:22.302148 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } attempt: 4
I20260621 02:41:22.303150 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.302668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60624)
0621 02:41:22.302877 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.302952 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 02:41:22.303335 13951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:22.304100 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.303830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60636)
0621 02:41:22.303942 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.304006 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260621 02:41:22.304378 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
I20260621 02:41:22.304579 13951 hybrid_clock.cc:648] HybridClock initialized: now 1782009682304520 us; error 48 us; skew 500 ppm
I20260621 02:41:22.304908 13951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:22.308117 13951 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260621 02:41:22.309664 13951 webserver.cc:460] Webserver started at http://127.0.97.60:43267/ using document root <none> and password file <none>
I20260621 02:41:22.310322 13951 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:22.310464 13951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:22.310748 13951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:22.313899 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/data/instance:
uuid: "4fc8c6142cf04fa1927b327de2b2b277"
format_stamp: "Formatted at 2026-06-21 02:41:22 on dist-test-slave-jkn7"
server_key: "e144c87205b89a83497f4a487787c635"
server_key_iv: "e87be86a65e85e566baed085abacc2ba"
server_key_version: "encryptionkey@0"
I20260621 02:41:22.314666 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/wal/instance:
uuid: "4fc8c6142cf04fa1927b327de2b2b277"
format_stamp: "Formatted at 2026-06-21 02:41:22 on dist-test-slave-jkn7"
server_key: "e144c87205b89a83497f4a487787c635"
server_key_iv: "e87be86a65e85e566baed085abacc2ba"
server_key_version: "encryptionkey@0"
I20260621 02:41:22.320749 13951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260621 02:41:22.324117 13966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:22.325487 13951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:41:22.325670 13951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/wal
uuid: "4fc8c6142cf04fa1927b327de2b2b277"
format_stamp: "Formatted at 2026-06-21 02:41:22 on dist-test-slave-jkn7"
server_key: "e144c87205b89a83497f4a487787c635"
server_key_iv: "e87be86a65e85e566baed085abacc2ba"
server_key_version: "encryptionkey@0"
I20260621 02:41:22.325856 13951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:22.344429 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } attempt: 3
I20260621 02:41:22.345628 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.345069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60646)
0621 02:41:22.345265 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.345419 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 02:41:22.346644 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.346260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60660)
0621 02:41:22.346456 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.346537 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260621 02:41:22.346856 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39289: connect: Connection refused (error 111)
I20260621 02:41:22.347450 13951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:22.351568 13951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:22.351881 13951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:22.370407 13951 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39289
I20260621 02:41:22.370450 14017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39289 every 8 connection(s)
I20260621 02:41:22.371663 13951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
I20260621 02:41:22.372838 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 13951
I20260621 02:41:22.373027 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009483336832-388-0/minicluster-data/master-2/wal/instance
I20260621 02:41:22.373642 388 external_mini_cluster.cc:1428] Setting key cb6ee2582f92b0a9635560625dadec1f
I20260621 02:41:22.376310 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:22.384990 14018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:22.398366 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.375511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.97.60:39289)
0621 02:41:22.377230 (+ 1719us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.377242 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.377332 (+ 90us) server_negotiation.cc:407] Connection header received
0621 02:41:22.378898 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.378918 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.379278 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.379596 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.381455 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.382752 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.384614 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.385247 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.385447 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.390074 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.390083 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.390097 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.390140 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.393746 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.394618 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.394625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.394630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.394847 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.395344 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.395351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.395354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.396025 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.396329 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.396735 (+ 406us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.397219 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1520,"thread_start_us":1291,"threads_started":1}
I20260621 02:41:22.404798 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.387983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35155 (local address 127.0.0.1:35066)
0621 02:41:22.388766 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.388815 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:22.388973 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:22.389441 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:22.389446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:22.390084 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:22.390467 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.390481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.391613 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.391622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.393385 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.393401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.394331 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.394338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.394507 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.395119 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:22.395135 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:22.397798 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:22.401726 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.401740 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.401747 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.402288 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.402811 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.402817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.402820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.402976 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.403710 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:22.403718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:22.403907 (+ 189us) client_negotiation.cc:769] Sending connection context
0621 02:41:22.404106 (+ 199us) client_negotiation.cc:240] Negotiation successful
0621 02:41:22.404419 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":431,"threads_started":1}
I20260621 02:41:22.405478 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.387999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.0.97.62:35155)
0621 02:41:22.388224 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.388232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.388926 (+ 694us) server_negotiation.cc:407] Connection header received
0621 02:41:22.389164 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.389170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.389250 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.389386 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.390630 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.391458 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.393526 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.394113 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.394252 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.398008 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.398015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.398020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.398058 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.401558 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.402427 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.402436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.402442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.402667 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.403123 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.403131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.403136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.403550 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.403740 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.404278 (+ 538us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.404571 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:41:22.406680 14018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:22.420606 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.407323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37087 (local address 127.0.0.1:35414)
0621 02:41:22.407549 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.407571 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:22.407673 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:22.407959 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:22.407964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:22.408379 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:22.408540 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.408550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.409905 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.409909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.411155 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.411166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.412058 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.412063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.412206 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.412713 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:22.412729 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:22.414621 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:22.418094 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.418100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.418104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.418525 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.418950 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.418955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.418958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.419133 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.419724 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:22.419728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:22.419873 (+ 145us) client_negotiation.cc:769] Sending connection context
0621 02:41:22.420019 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 02:41:22.420273 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260621 02:41:22.420989 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.401515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.0.97.62:35155)
0621 02:41:22.405706 (+ 4191us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.405712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.405726 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:41:22.405768 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.405772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.405834 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.405930 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.407422 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.408580 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.410242 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.410852 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.410998 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.415620 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.415625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.415628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.415655 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.418187 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.418960 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.418966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.418969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.419119 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.419550 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.419554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.419556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.419876 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.420162 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.420469 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.420667 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4082,"thread_start_us":3988,"threads_started":1}
I20260621 02:41:22.421111 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.407388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.0.97.61:37087)
0621 02:41:22.407550 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.407554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.407637 (+ 83us) server_negotiation.cc:407] Connection header received
0621 02:41:22.407771 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.407774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.407829 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.407906 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.408700 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.409746 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.411277 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.411886 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.412016 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.414797 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.414804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.414809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.414846 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.417964 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.418644 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.418650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.418654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.418829 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.419241 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.419245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.419248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.419614 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.419726 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.420124 (+ 398us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.420359 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:41:22.422060 14018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260621 02:41:22.435076 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.423121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.0.97.60:39289)
0621 02:41:22.423271 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.423275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.423287 (+ 12us) server_negotiation.cc:407] Connection header received
0621 02:41:22.423318 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.423321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.423364 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.423431 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.424311 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.424944 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.426108 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.426502 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.426636 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.429650 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.429656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.429660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.429713 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.432457 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.433043 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.433050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.433053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.433180 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.433557 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.433565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.433568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.433858 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.433994 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.434382 (+ 388us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.434737 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":55}
I20260621 02:41:22.435076 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.422692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60690)
0621 02:41:22.422845 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.422863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:22.422957 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:22.423505 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:22.423511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:22.423992 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:22.424178 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.424190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.425062 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.425070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.425980 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.425990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.426645 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.426650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.426863 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.427394 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:22.427410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:22.429489 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:22.432575 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.432583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.432587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.432937 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.433332 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.433337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.433340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.433457 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.433970 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:22.433974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:22.434103 (+ 129us) client_negotiation.cc:769] Sending connection context
0621 02:41:22.434299 (+ 196us) client_negotiation.cc:240] Negotiation successful
0621 02:41:22.434587 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":2}
I20260621 02:41:22.437557 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.422575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.0.97.61:37087)
0621 02:41:22.422711 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.422718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.422738 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:41:22.422924 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.422929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.422995 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.423096 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.424588 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.425261 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.426481 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.426878 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.427017 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.431466 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.431474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.431482 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.431521 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.434193 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.434843 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.434850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.434853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.434982 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.436094 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.436097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.436099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.436416 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.436690 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.437004 (+ 314us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.437209 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:41:22.440495 14018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277: Bootstrap starting.
I20260621 02:41:22.442025 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:41:22.443898 14018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:22.444973 14018 log.cc:824] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:22.448040 14018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277: No bootstrap required, opened a new log
I20260621 02:41:22.470721 14018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } }
I20260621 02:41:22.471180 14018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:22.471302 14018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc8c6142cf04fa1927b327de2b2b277, State: Initialized, Role: FOLLOWER
I20260621 02:41:22.472110 14018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } }
I20260621 02:41:22.477106 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.447631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.0.97.60:39289)
0621 02:41:22.453521 (+ 5890us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.453530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.457654 (+ 4124us) server_negotiation.cc:407] Connection header received
0621 02:41:22.457916 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.457925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.458022 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.458156 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.460981 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.462449 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.465326 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.465950 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.466089 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.469919 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.469925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.469930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.469966 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.473582 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.474446 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.474451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.474455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.474664 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.475160 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.475165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.475168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.475592 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.475852 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.476491 (+ 639us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.476718 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5700}
I20260621 02:41:22.479127 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.479511 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:41:22.481029 14018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:41:22.484010 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.447434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.0.97.61:37087)
0621 02:41:22.447859 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.447867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.465888 (+ 18021us) server_negotiation.cc:407] Connection header received
0621 02:41:22.466091 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.466096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.466178 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.466286 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.469306 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.470380 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.472633 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.473325 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.473429 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.477096 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.477103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.477109 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.477151 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.480691 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.481509 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.481515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.481518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.481697 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.482168 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.482175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.482180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.482529 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.482788 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.483261 (+ 473us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.483550 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260621 02:41:22.485460 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.446845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.0.97.62:35155)
0621 02:41:22.447028 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.447034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.466450 (+ 19416us) server_negotiation.cc:407] Connection header received
0621 02:41:22.466590 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.466595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.466677 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.466793 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.468796 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.469908 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.472162 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.472900 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.473026 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.478820 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.478828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.478835 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.478878 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.481970 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.482857 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.482864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.482869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.483027 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.483550 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.483556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.483561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.483945 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.484708 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.484846 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.485070 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 02:41:22.493367 14043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:41:22.493489 14043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:41:22.490203 13951 master_runner.cc:186] Error getting master registration for 127.0.97.62:35155: OK, instance_id { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" instance_seqno: 1782009681389141 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:41:22.499568 13951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:41:22.524092 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } attempt: 4
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260621 02:41:22.540004 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.524765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.0.97.60:39289)
0621 02:41:22.524943 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.524949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.524972 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:41:22.525100 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.525105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.525219 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.525377 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.526146 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.527388 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.529596 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.530133 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.530236 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.533941 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.533946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.533950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.533979 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.536974 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.537742 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.537749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.537752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.537925 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.538393 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.538398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.538402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.538793 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.538936 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.539413 (+ 477us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.539626 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:41:22.540022 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.524669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60720)
0621 02:41:22.524877 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.524894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:22.524982 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:22.525390 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:22.525394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:22.525769 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:22.525991 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.526000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.527814 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.527821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.529418 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.529434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.530339 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.530346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.530543 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.531153 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:22.531177 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:22.533763 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:22.537122 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.537129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.537133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.537598 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.538080 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.538084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.538088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.538283 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.539000 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:22.539005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:22.539165 (+ 160us) client_negotiation.cc:769] Sending connection context
0621 02:41:22.539307 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 02:41:22.539622 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 02:41:22.544441 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21: Bootstrap starting.
I20260621 02:41:22.547633 13870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:22.548619 13870 log.cc:824] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:22.551831 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21: No bootstrap required, opened a new log
I20260621 02:41:22.558642 13870 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } }
I20260621 02:41:22.558908 13870 raft_consensus.cc:374] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:22.559069 13870 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee77737809b4a6b8c7d1ba0ba4eab21, State: Initialized, Role: FOLLOWER
I20260621 02:41:22.559603 13870 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } }
I20260621 02:41:22.560474 14046 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.560848 14046 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:41:22.561688 13870 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:41:22.572772 14057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:41:22.572911 14057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:41:22.571873 13803 master_runner.cc:186] Error getting master registration for 127.0.97.61:37087: OK, instance_id { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" instance_seqno: 1782009681880726 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:41:22.584043 13803 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:41:22.603631 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } attempt: 5
W20260621 02:41:22.604043 13972 tablet.cc:2369] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:41:22 dist-test-slave-jkn7 krb5kdc[13790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260621 02:41:22.622696 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.604257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39289 (local address 127.0.0.1:60732)
0621 02:41:22.604634 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:22.604654 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:22.604764 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:22.605551 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:22.605558 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:22.606065 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:22.606285 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.606297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.607755 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.607761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.609328 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:22.609344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.611821 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.611827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:22.612041 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.612757 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:22.612781 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:22.615666 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:22.619833 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.619840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.619845 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.620376 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.620799 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:22.620803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:22.620806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:22.620966 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:22.621663 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:22.621668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:22.621832 (+ 164us) client_negotiation.cc:769] Sending connection context
0621 02:41:22.621991 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 02:41:22.622288 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260621 02:41:22.623054 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:22.604974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.0.97.60:39289)
0621 02:41:22.605177 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:22.605184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:22.605208 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:41:22.605268 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:22.605274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:22.605375 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:22.605486 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:22.606440 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.607602 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.610894 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:22.611658 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:22.611783 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:22.615866 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:22.615873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:22.615878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:22.615926 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:22.619688 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.620502 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.620508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.620512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.620672 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:22.621089 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:22.621094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:22.621098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:22.621507 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:22.621669 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:22.622110 (+ 441us) server_negotiation.cc:299] Negotiation successful
0621 02:41:22.622366 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 02:41:22.628911 13824 tablet.cc:2369] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:41:22.629709 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab: Bootstrap starting.
I20260621 02:41:22.633818 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:22.635092 13944 log.cc:824] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:22.638273 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab: No bootstrap required, opened a new log
I20260621 02:41:22.649034 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } }
I20260621 02:41:22.649479 13944 raft_consensus.cc:374] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:22.649603 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65939eb8af8443aa6976b9fad779bab, State: Initialized, Role: FOLLOWER
I20260621 02:41:22.650303 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } }
I20260621 02:41:22.653825 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.654279 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:41:22.655117 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:41:22.669589 13877 master_runner.cc:263] No leader master found from master b65939eb8af8443aa6976b9fad779bab
I20260621 02:41:22.669734 13877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260621 02:41:22.674003 14071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:41:22.674216 14071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:41:22.749583 14046 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:22.749908 14046 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } }
I20260621 02:41:22.752708 14046 leader_election.cc:288] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65939eb8af8443aa6976b9fad779bab (127.0.97.61:37087), 4fc8c6142cf04fa1927b327de2b2b277 (127.0.97.60:39289)
I20260621 02:41:22.753959 13919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65939eb8af8443aa6976b9fad779bab" is_pre_election: true
I20260621 02:41:22.754599 13919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee77737809b4a6b8c7d1ba0ba4eab21 in term 0.
I20260621 02:41:22.755455 13819 leader_election.cc:302] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65939eb8af8443aa6976b9fad779bab, eee77737809b4a6b8c7d1ba0ba4eab21; no voters:
I20260621 02:41:22.756582 13993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc8c6142cf04fa1927b327de2b2b277" is_pre_election: true
I20260621 02:41:22.757225 13993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee77737809b4a6b8c7d1ba0ba4eab21 in term 0.
I20260621 02:41:22.758308 14046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:41:22.758455 14046 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:41:22.758555 14046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:22.762146 14046 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } }
I20260621 02:41:22.763310 13919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65939eb8af8443aa6976b9fad779bab"
I20260621 02:41:22.763521 13919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:22.766953 13919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee77737809b4a6b8c7d1ba0ba4eab21 in term 1.
I20260621 02:41:22.767473 13819 leader_election.cc:302] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65939eb8af8443aa6976b9fad779bab, eee77737809b4a6b8c7d1ba0ba4eab21; no voters:
I20260621 02:41:22.774757 13993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc8c6142cf04fa1927b327de2b2b277"
I20260621 02:41:22.774983 13993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:22.778544 13993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee77737809b4a6b8c7d1ba0ba4eab21 in term 1.
I20260621 02:41:22.779244 14046 leader_election.cc:288] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [CANDIDATE]: Term 1 election: Requested vote from peers b65939eb8af8443aa6976b9fad779bab (127.0.97.61:37087), 4fc8c6142cf04fa1927b327de2b2b277 (127.0.97.60:39289)
I20260621 02:41:22.779616 14046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:41:22.780023 14046 raft_consensus.cc:686] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [term 1 LEADER]: Becoming Leader. State: Replica: eee77737809b4a6b8c7d1ba0ba4eab21, State: Running, Role: LEADER
I20260621 02:41:22.781577 14046 consensus_queue.cc:237] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } }
I20260621 02:41:22.800915 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee77737809b4a6b8c7d1ba0ba4eab21. Latest consensus state: current_term: 1 leader_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.801479 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:22.809804 14075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:41:22.812047 14075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:41:22.827914 13993 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [term 1 FOLLOWER]: Refusing update from remote peer eee77737809b4a6b8c7d1ba0ba4eab21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:22.828946 14073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:41:22.836634 13919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [term 1 FOLLOWER]: Refusing update from remote peer eee77737809b4a6b8c7d1ba0ba4eab21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:22.837549 14073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:41:22.863602 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee77737809b4a6b8c7d1ba0ba4eab21. Latest consensus state: current_term: 1 leader_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.863915 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:41:22.867790 14076 mvcc.cc:205] Tried to move back new op lower bound from 7299111660849102848 to 7299111660685402112. Current Snapshot: MvccSnapshot[applied={T|T < 7299111660849102848}]
I20260621 02:41:22.871672 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.871903 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:41:22.873853 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.874135 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:22.876288 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee77737809b4a6b8c7d1ba0ba4eab21. Latest consensus state: current_term: 1 leader_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.876569 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:41:22.876880 14075 catalog_manager.cc:1347] Generated new cluster ID: 2e6369e985074d94b6412961c073bbe5
I20260621 02:41:22.876961 14075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:41:22.896113 14046 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.896509 14046 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:22.899766 14077 mvcc.cc:205] Tried to move back new op lower bound from 7299111660849102848 to 7299111660685402112. Current Snapshot: MvccSnapshot[applied={T|T < 7299111660849102848}]
I20260621 02:41:22.916647 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee77737809b4a6b8c7d1ba0ba4eab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35155 } } peers { permanent_uuid: "b65939eb8af8443aa6976b9fad779bab" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37087 } } peers { permanent_uuid: "4fc8c6142cf04fa1927b327de2b2b277" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39289 } } }
I20260621 02:41:22.916936 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:41:22.918274 14075 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:41:22.919945 14075 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:41:22.935483 14075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eee77737809b4a6b8c7d1ba0ba4eab21: Generated new TSK 0
I20260621 02:41:22.936686 14075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:41:23.495429 14043 catalog_manager.cc:1259] Loaded cluster ID: 2e6369e985074d94b6412961c073bbe5
I20260621 02:41:23.495533 14043 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277: loading cluster ID for follower catalog manager: success
I20260621 02:41:23.498157 14043 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277: acquiring CA information for follower catalog manager: success
I20260621 02:41:23.499787 14043 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fc8c6142cf04fa1927b327de2b2b277: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:41:23.676291 14071 catalog_manager.cc:1259] Loaded cluster ID: 2e6369e985074d94b6412961c073bbe5
I20260621 02:41:23.676401 14071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab: loading cluster ID for follower catalog manager: success
I20260621 02:41:23.678836 14071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab: acquiring CA information for follower catalog manager: success
I20260621 02:41:23.680531 14071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b65939eb8af8443aa6976b9fad779bab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:41:24.029628 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 13803
I20260621 02:41:24.030937 13865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:24.147363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 13803
I20260621 02:41:24.168630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 13877
I20260621 02:41:24.169781 13939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:24.276827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 13877
I20260621 02:41:24.296965 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 13951
I20260621 02:41:24.298123 14013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:24.405489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 13951
2026-06-21T02:41:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4769 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:41:24 dist-test-slave-jkn7 krb5kdc[14093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:41:24 dist-test-slave-jkn7 krb5kdc[14093](info): set up 2 sockets
Jun 21 02:41:24 dist-test-slave-jkn7 krb5kdc[14093](info): commencing operation
krb5kdc: starting...
W20260621 02:41:25.517892 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:41:25 dist-test-slave-jkn7 krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:41:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:41:25.726135 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37749
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:26.044338 14106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:26.044656 14106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:26.044812 14106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:26.051884 14106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:26.051985 14106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:26.052030 14106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:26.052079 14106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:26.052124 14106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:26.061652 14106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:26.062827 14106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:26.064030 14106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:26.075333 14111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:26.075381 14112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:26.075381 14114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:26.076344 14106 server_base.cc:1034] running on GCE node
I20260621 02:41:26.077065 14106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:26.078465 14106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:26.079694 14106 hybrid_clock.cc:648] HybridClock initialized: now 1782009686079666 us; error 45 us; skew 500 ppm
I20260621 02:41:26.080036 14106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:26 dist-test-slave-jkn7 krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:26.083057 14106 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:41:26.084513 14106 webserver.cc:460] Webserver started at http://127.0.97.62:36053/ using document root <none> and password file <none>
I20260621 02:41:26.085173 14106 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:26.085362 14106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:26.085673 14106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:26.088479 14106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "c23c5b31f52745cfa9959155c1e02e15"
format_stamp: "Formatted at 2026-06-21 02:41:26 on dist-test-slave-jkn7"
server_key: "52331230a9e17fe0bf3b276bedf3c8a3"
server_key_iv: "6c776c5904e8bca9e27d003c0993e648"
server_key_version: "encryptionkey@0"
I20260621 02:41:26.089213 14106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "c23c5b31f52745cfa9959155c1e02e15"
format_stamp: "Formatted at 2026-06-21 02:41:26 on dist-test-slave-jkn7"
server_key: "52331230a9e17fe0bf3b276bedf3c8a3"
server_key_iv: "6c776c5904e8bca9e27d003c0993e648"
server_key_version: "encryptionkey@0"
I20260621 02:41:26.093984 14106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:41:26.097390 14121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:26.098592 14106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:41:26.098781 14106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "c23c5b31f52745cfa9959155c1e02e15"
format_stamp: "Formatted at 2026-06-21 02:41:26 on dist-test-slave-jkn7"
server_key: "52331230a9e17fe0bf3b276bedf3c8a3"
server_key_iv: "6c776c5904e8bca9e27d003c0993e648"
server_key_version: "encryptionkey@0"
I20260621 02:41:26.098974 14106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:26.124367 14106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:26.128520 14106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:26.128815 14106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:26.147555 14106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37749
I20260621 02:41:26.147603 14172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37749 every 8 connection(s)
I20260621 02:41:26.148891 14106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:41:26.152606 14173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:26.156818 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 14106
I20260621 02:41:26.157047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:41:26.157768 388 external_mini_cluster.cc:1428] Setting key 7819381a83cb55ca95110d41c7d9e289
I20260621 02:41:26.163467 14173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15: Bootstrap starting.
I20260621 02:41:26.167066 14173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:26.168282 14173 log.cc:824] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:26.171657 14173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15: No bootstrap required, opened a new log
Jun 21 02:41:26 dist-test-slave-jkn7 krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:26.179747 14173 raft_consensus.cc:348] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23c5b31f52745cfa9959155c1e02e15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37749 } }
I20260621 02:41:26.180150 14173 raft_consensus.cc:374] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:26.180275 14173 raft_consensus.cc:729] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23c5b31f52745cfa9959155c1e02e15, State: Initialized, Role: FOLLOWER
I20260621 02:41:26.180893 14173 consensus_queue.cc:260] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23c5b31f52745cfa9959155c1e02e15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37749 } }
I20260621 02:41:26.181141 14173 raft_consensus.cc:388] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:41:26.181336 14173 raft_consensus.cc:482] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:41:26.181524 14173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:26.181694 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:26.159594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.0.97.62:37749)
0621 02:41:26.160437 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:26.160449 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:26.160488 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:41:26.162289 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:26.162313 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:26.162687 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:26.162987 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:26.164848 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:26.166262 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:26.168157 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:26.169061 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:26.169265 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:26.174039 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:26.174047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:26.174061 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:26.174101 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:26.177159 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:26.177950 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:26.177956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:26.177965 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:26.178147 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:26.178558 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:26.178565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:26.178573 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:26.179090 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:26.179405 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:26.179876 (+ 471us) server_negotiation.cc:299] Negotiation successful
0621 02:41:26.180368 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":377,"threads_started":1}
I20260621 02:41:26.183944 14173 raft_consensus.cc:504] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23c5b31f52745cfa9959155c1e02e15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37749 } }
I20260621 02:41:26.184500 14173 leader_election.cc:302] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c23c5b31f52745cfa9959155c1e02e15; no voters:
I20260621 02:41:26.185249 14173 leader_election.cc:288] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:41:26.185376 14178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:41:26.185532 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:41:26.187760 14173 sys_catalog.cc:564] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:41:26.187944 14178 raft_consensus.cc:686] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [term 1 LEADER]: Becoming Leader. State: Replica: c23c5b31f52745cfa9959155c1e02e15, State: Running, Role: LEADER
I20260621 02:41:26.188572 14178 consensus_queue.cc:237] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23c5b31f52745cfa9959155c1e02e15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37749 } }
I20260621 02:41:26.191869 14180 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c23c5b31f52745cfa9959155c1e02e15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23c5b31f52745cfa9959155c1e02e15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37749 } } }
I20260621 02:41:26.192237 14180 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:26.192752 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c23c5b31f52745cfa9959155c1e02e15. Latest consensus state: current_term: 1 leader_uuid: "c23c5b31f52745cfa9959155c1e02e15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23c5b31f52745cfa9959155c1e02e15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37749 } } }
I20260621 02:41:26.192982 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:26.194447 14191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:41:26.201217 14191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:41:26.209156 14191 catalog_manager.cc:1347] Generated new cluster ID: 8117563ebaec4587af20143c7aa5b959
I20260621 02:41:26.209263 14191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:41:26.240135 14191 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:41:26.241516 14191 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:41:26.252774 14191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c23c5b31f52745cfa9959155c1e02e15: Generated new TSK 0
I20260621 02:41:26.253602 14191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:41:26.298164 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 14106
I20260621 02:41:26.299469 14168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:26.398288 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 14106
2026-06-21T02:41:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1988 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:41:26 dist-test-slave-jkn7 krb5kdc[14201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:41:26 dist-test-slave-jkn7 krb5kdc[14201](info): set up 2 sockets
Jun 21 02:41:26 dist-test-slave-jkn7 krb5kdc[14201](info): commencing operation
krb5kdc: starting...
W20260621 02:41:27.497782 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:41:27 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:41:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:41:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:41:27.712527 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34379
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:28.029357 14214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:28.029716 14214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:28.029878 14214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:28.036945 14214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:28.037078 14214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:28.037181 14214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:28.037317 14214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:28.037398 14214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:28.046936 14214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:28.048228 14214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:28.049562 14214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:28.061169 14219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:28.061218 14220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:28.061240 14222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:28.062158 14214 server_base.cc:1034] running on GCE node
I20260621 02:41:28.062966 14214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:28.064411 14214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:28.065635 14214 hybrid_clock.cc:648] HybridClock initialized: now 1782009688065586 us; error 40 us; skew 500 ppm
I20260621 02:41:28.065971 14214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:28 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:28.069020 14214 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:41:28.070636 14214 webserver.cc:460] Webserver started at http://127.0.97.62:34387/ using document root <none> and password file <none>
I20260621 02:41:28.071339 14214 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:28.071487 14214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:28.071800 14214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:28.074855 14214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "a7285edd9e6b46c8a852e9e63e4a85a3"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "d74d8f9adaed7084d364e38557051540"
server_key_iv: "fd61f0fc268469b98c8bf4e6efa26cc5"
server_key_version: "encryptionkey@0"
I20260621 02:41:28.075631 14214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "a7285edd9e6b46c8a852e9e63e4a85a3"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "d74d8f9adaed7084d364e38557051540"
server_key_iv: "fd61f0fc268469b98c8bf4e6efa26cc5"
server_key_version: "encryptionkey@0"
I20260621 02:41:28.080739 14214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:41:28.084045 14229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:28.085465 14214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:41:28.085649 14214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "a7285edd9e6b46c8a852e9e63e4a85a3"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "d74d8f9adaed7084d364e38557051540"
server_key_iv: "fd61f0fc268469b98c8bf4e6efa26cc5"
server_key_version: "encryptionkey@0"
I20260621 02:41:28.085855 14214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:28.100782 14214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:28.104669 14214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:28.104949 14214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:28.123749 14214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34379
I20260621 02:41:28.123833 14280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34379 every 8 connection(s)
I20260621 02:41:28.125092 14214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:41:28.128679 14281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:28.131906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 14214
I20260621 02:41:28.132134 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:41:28.132761 388 external_mini_cluster.cc:1428] Setting key fd67a5b0f0c75aaef94ec9af7d2f3f6a
I20260621 02:41:28.141635 14281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Bootstrap starting.
I20260621 02:41:28.145097 14281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:28.146097 14281 log.cc:824] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:41:28 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:28.149202 14281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: No bootstrap required, opened a new log
I20260621 02:41:28.156736 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:28.134692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.0.97.62:34379)
0621 02:41:28.135504 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:28.135518 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:28.135566 (+ 48us) server_negotiation.cc:407] Connection header received
0621 02:41:28.137471 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:28.137501 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:28.138019 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:28.138435 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:28.139730 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:28.141680 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:28.143031 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:28.144055 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:28.144246 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:28.148433 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:28.148443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:28.148459 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:28.148499 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:28.152435 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:28.153125 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:28.153133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:28.153141 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:28.153380 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:28.153747 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:28.153752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:28.153755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:28.154413 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:28.154725 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:28.155043 (+ 318us) server_negotiation.cc:299] Negotiation successful
0621 02:41:28.155448 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":330,"threads_started":1}
I20260621 02:41:28.159682 14281 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:28.159976 14281 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:28.160050 14281 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7285edd9e6b46c8a852e9e63e4a85a3, State: Initialized, Role: FOLLOWER
I20260621 02:41:28.160576 14281 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:28.160765 14281 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:41:28.160859 14281 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:41:28.160974 14281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:28.163282 14281 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:28.163686 14281 leader_election.cc:302] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7285edd9e6b46c8a852e9e63e4a85a3; no voters:
I20260621 02:41:28.164197 14281 leader_election.cc:288] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:41:28.164331 14286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:41:28.165027 14286 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 1 LEADER]: Becoming Leader. State: Replica: a7285edd9e6b46c8a852e9e63e4a85a3, State: Running, Role: LEADER
I20260621 02:41:28.165613 14286 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:28.166237 14281 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:41:28.168303 14287 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } } }
I20260621 02:41:28.168617 14287 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:28.168996 14288 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7285edd9e6b46c8a852e9e63e4a85a3. Latest consensus state: current_term: 1 leader_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } } }
I20260621 02:41:28.169260 14288 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:28.171276 14292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:41:28.176648 14292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:41:28.184893 14292 catalog_manager.cc:1347] Generated new cluster ID: 8b901cf13c0044b0816dd3db90914fdf
I20260621 02:41:28.185029 14292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:41:28.216708 14292 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:41:28.218125 14292 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:41:28.225757 14292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Generated new TSK 0
I20260621 02:41:28.226550 14292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:41:28.292229 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34379
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:28.623881 14307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:28.624217 14307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:28.624372 14307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:28.631616 14307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:28.631839 14307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:28.631994 14307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:41:28.643049 14307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:28.644403 14307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:28.645841 14307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:28.659922 14315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:28.660007 14312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:28.659973 14313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:28.660749 14307 server_base.cc:1034] running on GCE node
I20260621 02:41:28.661396 14307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:28.662541 14307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:28.663738 14307 hybrid_clock.cc:648] HybridClock initialized: now 1782009688663693 us; error 46 us; skew 500 ppm
I20260621 02:41:28.664000 14307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:28 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:28.667299 14307 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:41:28.668766 14307 webserver.cc:460] Webserver started at http://127.0.97.1:33269/ using document root <none> and password file <none>
I20260621 02:41:28.669533 14307 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:28.669678 14307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:28.669957 14307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:28.673101 14307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "83ae2e3426cb4de585d9c2297b823c8c"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "ef3aad1a257124edd81461a0f17d711e"
server_key_iv: "5aa320b60498e5b2e67a9e50a4cc5b18"
server_key_version: "encryptionkey@0"
I20260621 02:41:28.673884 14307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "83ae2e3426cb4de585d9c2297b823c8c"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "ef3aad1a257124edd81461a0f17d711e"
server_key_iv: "5aa320b60498e5b2e67a9e50a4cc5b18"
server_key_version: "encryptionkey@0"
I20260621 02:41:28.678725 14307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 02:41:28.681906 14322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:28.683139 14307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:41:28.683277 14307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "83ae2e3426cb4de585d9c2297b823c8c"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "ef3aad1a257124edd81461a0f17d711e"
server_key_iv: "5aa320b60498e5b2e67a9e50a4cc5b18"
server_key_version: "encryptionkey@0"
I20260621 02:41:28.683425 14307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:28.701956 14307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:28.705994 14307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:28.706231 14307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:28.707083 14307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:28.708283 14307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:28.708355 14307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:28.708438 14307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:28.708482 14307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:28.741178 14307 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33399
I20260621 02:41:28.741211 14434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33399 every 8 connection(s)
I20260621 02:41:28.742451 14307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:41:28.746498 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 14307
I20260621 02:41:28.746727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:41:28.747351 388 external_mini_cluster.cc:1428] Setting key c51087300f5b0ec7f23e4b8adb575b34
Jun 21 02:41:28 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:28.762892 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:28.744885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56675 (local address 127.0.97.62:34379)
0621 02:41:28.745395 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:28.745399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:28.746204 (+ 805us) server_negotiation.cc:407] Connection header received
0621 02:41:28.748170 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:28.748176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:28.748259 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:28.748381 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:28.750460 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:28.751652 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:28.753122 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:28.753898 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:28.754049 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:28.757453 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:28.757458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:28.757462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:28.757492 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:28.759965 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:28.760642 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:28.760649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:28.760657 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:28.760771 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:28.761195 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:28.761206 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:28.761209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:28.761533 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:28.761683 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:28.762333 (+ 650us) server_negotiation.cc:299] Negotiation successful
0621 02:41:28.762553 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":248,"threads_started":1}
I20260621 02:41:28.763712 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:28.745274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:56675)
0621 02:41:28.746043 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:28.746083 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:28.747826 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:28.749197 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:28.749210 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:28.749756 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:28.750283 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:28.750300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:28.751797 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:28.751801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:28.752966 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:28.752977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:28.754094 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:28.754099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:28.754314 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:28.754976 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:28.755001 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:28.757202 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:28.760102 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:28.760110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:28.760124 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:28.760513 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:28.760879 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:28.760885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:28.760889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:28.761068 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:28.761677 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:28.761684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:28.761971 (+ 287us) client_negotiation.cc:769] Sending connection context
0621 02:41:28.762217 (+ 246us) client_negotiation.cc:240] Negotiation successful
0621 02:41:28.762681 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":266,"threads_started":1}
I20260621 02:41:28.765172 14435 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34379
I20260621 02:41:28.765482 14435 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:28.766192 14435 heartbeater.cc:511] Master 127.0.97.62:34379 requested a full tablet report, sending...
I20260621 02:41:28.768273 14246 ts_manager.cc:194] Registered new tserver with Master: 83ae2e3426cb4de585d9c2297b823c8c (127.0.97.1:33399)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260621 02:41:28.769837 14246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:41:28.800822 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34379
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:29.125207 14441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:29.125566 14441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:29.125732 14441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:29.132752 14441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:29.132900 14441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:29.133070 14441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:41:29.142663 14441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:29.143966 14441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:29.145252 14441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:29.158340 14446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:29.158390 14449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:29.158620 14447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:29.159286 14441 server_base.cc:1034] running on GCE node
I20260621 02:41:29.159866 14441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:29.160763 14441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:29.162012 14441 hybrid_clock.cc:648] HybridClock initialized: now 1782009689161969 us; error 51 us; skew 500 ppm
I20260621 02:41:29.162358 14441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:29 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:29.165410 14441 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:41:29.166920 14441 webserver.cc:460] Webserver started at http://127.0.97.2:33513/ using document root <none> and password file <none>
I20260621 02:41:29.167623 14441 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:29.167768 14441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:29.168059 14441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:29.170917 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "0e0eddbc17954ae989cba507455893bb"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "fdf55df3496b8b4567427455897201fa"
server_key_iv: "f15ea3595df8f8d474f18c1d63dc436d"
server_key_version: "encryptionkey@0"
I20260621 02:41:29.171766 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0e0eddbc17954ae989cba507455893bb"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "fdf55df3496b8b4567427455897201fa"
server_key_iv: "f15ea3595df8f8d474f18c1d63dc436d"
server_key_version: "encryptionkey@0"
I20260621 02:41:29.176725 14441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:41:29.180236 14456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:29.181643 14441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:41:29.181772 14441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "0e0eddbc17954ae989cba507455893bb"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "fdf55df3496b8b4567427455897201fa"
server_key_iv: "f15ea3595df8f8d474f18c1d63dc436d"
server_key_version: "encryptionkey@0"
I20260621 02:41:29.181929 14441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:29.210834 14441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:29.214808 14441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:29.215049 14441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:29.215960 14441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:29.217211 14441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:29.217303 14441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:29.217388 14441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:29.217439 14441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:29.249500 14441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41737
I20260621 02:41:29.249588 14568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41737 every 8 connection(s)
I20260621 02:41:29.250741 14441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:41:29.252799 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 14441
I20260621 02:41:29.252974 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:41:29.253549 388 external_mini_cluster.cc:1428] Setting key d7df77d96341a16f4d685e7fa3582bd0
Jun 21 02:41:29 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:29.273825 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.253293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48889 (local address 127.0.97.62:34379)
0621 02:41:29.253500 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.253505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:29.254798 (+ 1293us) server_negotiation.cc:407] Connection header received
0621 02:41:29.257202 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:29.257207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:29.257334 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:29.257461 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:29.259487 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.260200 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.262691 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.263122 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.263219 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.267551 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:29.267556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:29.267559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:29.267590 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:29.270292 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:29.271264 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:29.271269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:29.271271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:29.271419 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:29.271946 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:29.271955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:29.271957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:29.272306 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:29.272464 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:29.273219 (+ 755us) server_negotiation.cc:299] Negotiation successful
0621 02:41:29.273474 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 02:41:29.274820 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.253858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:48889)
0621 02:41:29.254639 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:29.254682 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:29.256895 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:29.257716 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:29.257730 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:29.258505 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:29.259292 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.259319 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.260357 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.260365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.262530 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.262548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.263353 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.263362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.263713 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.264482 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:29.264510 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:29.267318 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:29.270439 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:29.270451 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:29.270470 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:29.271104 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:29.271592 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:29.271597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:29.271601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:29.271799 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:29.272436 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:29.272444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:29.272847 (+ 403us) client_negotiation.cc:769] Sending connection context
0621 02:41:29.273120 (+ 273us) client_negotiation.cc:240] Negotiation successful
0621 02:41:29.273658 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":354,"threads_started":1}
I20260621 02:41:29.276441 14569 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34379
I20260621 02:41:29.276748 14569 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:29.277511 14569 heartbeater.cc:511] Master 127.0.97.62:34379 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:41:29.279032 14246 ts_manager.cc:194] Registered new tserver with Master: 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737)
I20260621 02:41:29.279945 14246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:41:29.304522 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34379
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:29.624455 14574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:29.624730 14574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:29.624887 14574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:29.632656 14574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:29.632755 14574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:29.632962 14574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:41:29.642907 14574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:29.644239 14574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:29.645592 14574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:29.658349 14579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:29.658389 14582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:29.658704 14580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:29.659266 14574 server_base.cc:1034] running on GCE node
I20260621 02:41:29.659868 14574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:29.660820 14574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:29.662056 14574 hybrid_clock.cc:648] HybridClock initialized: now 1782009689661996 us; error 50 us; skew 500 ppm
I20260621 02:41:29.662392 14574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:29 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:29.665598 14574 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:41:29.667268 14574 webserver.cc:460] Webserver started at http://127.0.97.3:38093/ using document root <none> and password file <none>
I20260621 02:41:29.667922 14574 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:29.668022 14574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:29.668274 14574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:29.671226 14574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "a1ef4019af0548d492258bfa132feb33"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "86e3ba938e2060463b5909199e94a17b"
server_key_iv: "2493b1d2a48914760e3698d99bac4efb"
server_key_version: "encryptionkey@0"
I20260621 02:41:29.672103 14574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a1ef4019af0548d492258bfa132feb33"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "86e3ba938e2060463b5909199e94a17b"
server_key_iv: "2493b1d2a48914760e3698d99bac4efb"
server_key_version: "encryptionkey@0"
I20260621 02:41:29.677603 14574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:41:29.681110 14589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:29.682510 14574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:41:29.682650 14574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "a1ef4019af0548d492258bfa132feb33"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "86e3ba938e2060463b5909199e94a17b"
server_key_iv: "2493b1d2a48914760e3698d99bac4efb"
server_key_version: "encryptionkey@0"
I20260621 02:41:29.682822 14574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:29.699288 14574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:29.703377 14574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:29.703626 14574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:29.704500 14574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:29.705945 14574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:29.706018 14574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:29.706103 14574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:29.706148 14574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:29.739076 14574 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39251
I20260621 02:41:29.739118 14701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39251 every 8 connection(s)
I20260621 02:41:29.740283 14574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:41:29.746321 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 14574
I20260621 02:41:29.746522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:41:29.746977 388 external_mini_cluster.cc:1428] Setting key acc990b9a40a4a6c11732333b4be8b51
Jun 21 02:41:29 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:29.760365 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.742883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60531 (local address 127.0.97.62:34379)
0621 02:41:29.743096 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.743104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:29.744345 (+ 1241us) server_negotiation.cc:407] Connection header received
0621 02:41:29.746699 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:29.746704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:29.746800 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:29.746913 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:29.748592 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.749571 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.751159 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.751562 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.751692 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.755020 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:29.755025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:29.755028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:29.755060 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:29.757523 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:29.758186 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:29.758191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:29.758197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:29.758328 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:29.758733 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:29.758739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:29.758742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:29.758995 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:29.759148 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:29.759776 (+ 628us) server_negotiation.cc:299] Negotiation successful
0621 02:41:29.759994 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:41:29.761492 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.743349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:60531)
0621 02:41:29.744131 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:29.744178 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:29.746381 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:29.747148 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:29.747156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:29.747808 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:29.748429 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.748449 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.749719 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.749725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.751020 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.751031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.751806 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.751814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.752025 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.752691 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:29.752711 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:29.754826 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:29.757650 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:29.757658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:29.757672 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:29.758059 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:29.758445 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:29.758450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:29.758453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:29.758622 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:29.759120 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:29.759127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:29.759426 (+ 299us) client_negotiation.cc:769] Sending connection context
0621 02:41:29.759679 (+ 253us) client_negotiation.cc:240] Negotiation successful
0621 02:41:29.760154 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
I20260621 02:41:29.762884 14702 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34379
I20260621 02:41:29.763199 14702 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:29.763895 14702 heartbeater.cc:511] Master 127.0.97.62:34379 requested a full tablet report, sending...
I20260621 02:41:29.765254 14246 ts_manager.cc:194] Registered new tserver with Master: a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251)
I20260621 02:41:29.766227 14246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60531
I20260621 02:41:29.773007 14435 heartbeater.cc:503] Master 127.0.97.62:34379 was elected leader, sending a full tablet report...
I20260621 02:41:29.774765 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:41:29.791831 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.779311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.0.97.62:34379)
0621 02:41:29.779515 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.779525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:29.779861 (+ 336us) server_negotiation.cc:407] Connection header received
0621 02:41:29.779971 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:29.779974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:29.780029 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:29.780106 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:29.781578 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.782279 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.783653 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.784047 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.784135 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.786706 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:29.786711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:29.786714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:29.786749 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:29.789349 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:29.790094 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:29.790097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:29.790100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:29.790212 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:29.790525 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:29.790528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:29.790529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:29.790821 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:29.791000 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:29.791329 (+ 329us) server_negotiation.cc:299] Negotiation successful
0621 02:41:29.791484 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:41:29.795634 14245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:41:29.802407 14245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:41:29.842391 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.815015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.0.97.3:39251)
0621 02:41:29.832095 (+ 17080us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.832104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:29.832142 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:41:29.832250 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:29.832258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:29.832515 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:29.833562 (+ 1047us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:29.833885 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.836014 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.840016 (+ 4002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.841260 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.841418 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.841667 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:29.841915 (+ 248us) server_negotiation.cc:299] Negotiation successful
0621 02:41:29.842070 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16859,"thread_start_us":16680,"threads_started":1}
I20260621 02:41:29.842450 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.814891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39251 (local address 127.0.0.1:43658)
0621 02:41:29.829785 (+ 14894us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:29.829851 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:29.830098 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:29.832761 (+ 2663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:29.832769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:29.832800 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:29.833313 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.833337 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.836200 (+ 2863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.836209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.839469 (+ 3260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.839484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.841470 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.841476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.841659 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.841711 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 02:41:29.841854 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 02:41:29.842048 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14484,"mutex_wait_us":1,"thread_start_us":17156,"threads_started":1}
I20260621 02:41:29.843485 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.815018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41737 (local address 127.0.0.1:33146)
0621 02:41:29.832331 (+ 17313us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:29.832347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:29.832492 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:29.833497 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:29.833505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:29.833540 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:29.834135 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.834151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.836756 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.836764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.840621 (+ 3857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.840636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.842582 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.842588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.842760 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.842791 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:41:29.842861 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 02:41:29.842986 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17181,"mutex_wait_us":1596,"thread_start_us":15322,"threads_started":1}
I20260621 02:41:29.844485 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.815673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.0.97.2:41737)
0621 02:41:29.831098 (+ 15425us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.831111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:29.832457 (+ 1346us) server_negotiation.cc:407] Connection header received
0621 02:41:29.832725 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:29.832735 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:29.832949 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:29.833169 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:29.834397 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.836527 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.840822 (+ 4295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.842157 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.842281 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.842446 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:29.843179 (+ 733us) server_negotiation.cc:299] Negotiation successful
0621 02:41:29.843347 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15186,"thread_start_us":15428,"threads_started":1}
I20260621 02:41:29.846662 14637 tablet_service.cc:1467] Processing CreateTablet for tablet 3a881144963d488c8d96f5b4df89f8c4 (DEFAULT_TABLE table=test-table [id=5c16f1f8d9e049f3ba3b2f90b0d04ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:29.847254 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.832724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:43702)
0621 02:41:29.839292 (+ 6568us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:29.839310 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:29.839427 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:29.840644 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:29.840650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:29.840677 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:29.841045 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.841057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.843077 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.843084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.845031 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.845045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.846487 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.846494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.846669 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.846706 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:41:29.846822 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 02:41:29.846940 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6431,"thread_start_us":6775,"threads_started":1}
I20260621 02:41:29.847460 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.832841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.0.97.1:33399)
0621 02:41:29.840070 (+ 7229us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.840079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:29.840104 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:41:29.840205 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:29.840214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:29.840422 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:29.840627 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:29.841240 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.842946 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.845226 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.846291 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.846441 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.846657 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:29.846860 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 02:41:29.847014 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7075,"thread_start_us":6922,"threads_started":1}
I20260621 02:41:29.848248 14637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a881144963d488c8d96f5b4df89f8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:29.847986 14504 tablet_service.cc:1467] Processing CreateTablet for tablet 3a881144963d488c8d96f5b4df89f8c4 (DEFAULT_TABLE table=test-table [id=5c16f1f8d9e049f3ba3b2f90b0d04ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:29.849238 14504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a881144963d488c8d96f5b4df89f8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:29.850138 14370 tablet_service.cc:1467] Processing CreateTablet for tablet 3a881144963d488c8d96f5b4df89f8c4 (DEFAULT_TABLE table=test-table [id=5c16f1f8d9e049f3ba3b2f90b0d04ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:29.851158 14370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a881144963d488c8d96f5b4df89f8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:29.866030 14721 tablet_bootstrap.cc:492] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: Bootstrap starting.
I20260621 02:41:29.866187 14722 tablet_bootstrap.cc:492] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: Bootstrap starting.
I20260621 02:41:29.868049 14720 tablet_bootstrap.cc:492] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: Bootstrap starting.
I20260621 02:41:29.870221 14722 tablet_bootstrap.cc:654] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:29.870771 14721 tablet_bootstrap.cc:654] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:29.871336 14722 log.cc:824] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:29.871428 14720 tablet_bootstrap.cc:654] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:29.872125 14721 log.cc:824] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:29.872637 14720 log.cc:824] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:29.874536 14722 tablet_bootstrap.cc:492] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: No bootstrap required, opened a new log
I20260621 02:41:29.874809 14722 ts_tablet_manager.cc:1397] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260621 02:41:29.875877 14721 tablet_bootstrap.cc:492] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: No bootstrap required, opened a new log
I20260621 02:41:29.875874 14720 tablet_bootstrap.cc:492] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: No bootstrap required, opened a new log
I20260621 02:41:29.876138 14721 ts_tablet_manager.cc:1397] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260621 02:41:29.876134 14720 ts_tablet_manager.cc:1397] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260621 02:41:29.882896 14722 raft_consensus.cc:348] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260621 02:41:29.883338 14722 raft_consensus.cc:374] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:29.883447 14722 raft_consensus.cc:729] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0eddbc17954ae989cba507455893bb, State: Initialized, Role: FOLLOWER
I20260621 02:41:29.884112 14722 consensus_queue.cc:260] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260621 02:41:29.885388 14569 heartbeater.cc:503] Master 127.0.97.62:34379 was elected leader, sending a full tablet report...
I20260621 02:41:29.886292 14722 ts_tablet_manager.cc:1428] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260621 02:41:29.887449 14726 raft_consensus.cc:482] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:29.887228 14721 raft_consensus.cc:348] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260621 02:41:29.887745 14726 raft_consensus.cc:504] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260621 02:41:29.887869 14721 raft_consensus.cc:374] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:29.888083 14721 raft_consensus.cc:729] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ef4019af0548d492258bfa132feb33, State: Initialized, Role: FOLLOWER
I20260621 02:41:29.888592 14720 raft_consensus.cc:348] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260621 02:41:29.888819 14721 consensus_queue.cc:260] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260621 02:41:29.889045 14720 raft_consensus.cc:374] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:29.889207 14720 raft_consensus.cc:729] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ae2e3426cb4de585d9c2297b823c8c, State: Initialized, Role: FOLLOWER
I20260621 02:41:29.889933 14720 consensus_queue.cc:260] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260621 02:41:29.890606 14702 heartbeater.cc:503] Master 127.0.97.62:34379 was elected leader, sending a full tablet report...
I20260621 02:41:29.891356 14721 ts_tablet_manager.cc:1428] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260621 02:41:29.897513 14726 leader_election.cc:288] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251), 83ae2e3426cb4de585d9c2297b823c8c (127.0.97.1:33399)
I20260621 02:41:29.900154 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.890530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52113 (local address 127.0.97.3:39251)
0621 02:41:29.890853 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.890862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:29.892256 (+ 1394us) server_negotiation.cc:407] Connection header received
0621 02:41:29.892597 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:29.892603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:29.892673 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:29.892737 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:29.893996 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.895042 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.897678 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.898904 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.898995 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.899099 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:29.899759 (+ 660us) server_negotiation.cc:299] Negotiation successful
0621 02:41:29.899908 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260621 02:41:29.901062 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.889940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39251 (local address 127.0.97.2:52113)
0621 02:41:29.892146 (+ 2206us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:29.892167 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:29.892368 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:29.893406 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:29.893411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:29.893437 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:29.893828 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.893853 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.895227 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.895231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.897507 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.897520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.899333 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.899340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.899567 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.899608 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 02:41:29.900557 (+ 949us) client_negotiation.cc:240] Negotiation successful
0621 02:41:29.900705 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"thread_start_us":1889,"threads_started":1}
I20260621 02:41:29.901165 14720 ts_tablet_manager.cc:1428] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: Time spent starting tablet: real 0.025s user 0.019s sys 0.004s
I20260621 02:41:29.902423 14657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a881144963d488c8d96f5b4df89f8c4" candidate_uuid: "0e0eddbc17954ae989cba507455893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ef4019af0548d492258bfa132feb33" is_pre_election: true
I20260621 02:41:29.903008 14657 raft_consensus.cc:2454] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0eddbc17954ae989cba507455893bb in term 0.
I20260621 02:41:29.903718 14459 leader_election.cc:302] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0eddbc17954ae989cba507455893bb, a1ef4019af0548d492258bfa132feb33; no voters:
I20260621 02:41:29.904137 14726 raft_consensus.cc:2781] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:41:29.904294 14726 raft_consensus.cc:482] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:41:29.904464 14726 raft_consensus.cc:3037] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:29.905311 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.893240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58667 (local address 127.0.97.1:33399)
0621 02:41:29.893464 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.893470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:29.898441 (+ 4971us) server_negotiation.cc:407] Connection header received
0621 02:41:29.898682 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:29.898699 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:29.898778 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:29.898841 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:29.899692 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.900816 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.902895 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.904179 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.904324 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.904427 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:29.904897 (+ 470us) server_negotiation.cc:299] Negotiation successful
0621 02:41:29.905040 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260621 02:41:29.906010 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.893086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.97.2:58667)
0621 02:41:29.897376 (+ 4290us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:29.898364 (+ 988us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:29.898536 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:29.899035 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:29.899040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:29.899064 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:29.899487 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.899498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.901010 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.901015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.902711 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:29.902728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.904456 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.904465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:29.904676 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.904718 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 02:41:29.905526 (+ 808us) client_negotiation.cc:240] Negotiation successful
0621 02:41:29.905679 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4127,"thread_start_us":3966,"threads_started":1}
I20260621 02:41:29.906845 14390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a881144963d488c8d96f5b4df89f8c4" candidate_uuid: "0e0eddbc17954ae989cba507455893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ae2e3426cb4de585d9c2297b823c8c" is_pre_election: true
I20260621 02:41:29.907339 14390 raft_consensus.cc:2454] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0eddbc17954ae989cba507455893bb in term 0.
I20260621 02:41:29.907331 14726 raft_consensus.cc:504] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260621 02:41:29.907912 14726 leader_election.cc:288] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [CANDIDATE]: Term 1 election: Requested vote from peers a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251), 83ae2e3426cb4de585d9c2297b823c8c (127.0.97.1:33399)
I20260621 02:41:29.908330 14657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a881144963d488c8d96f5b4df89f8c4" candidate_uuid: "0e0eddbc17954ae989cba507455893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ef4019af0548d492258bfa132feb33"
I20260621 02:41:29.908569 14657 raft_consensus.cc:3037] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:29.908743 14390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a881144963d488c8d96f5b4df89f8c4" candidate_uuid: "0e0eddbc17954ae989cba507455893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ae2e3426cb4de585d9c2297b823c8c"
I20260621 02:41:29.908926 14390 raft_consensus.cc:3037] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:29.910944 14657 raft_consensus.cc:2454] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0eddbc17954ae989cba507455893bb in term 1.
I20260621 02:41:29.911115 14390 raft_consensus.cc:2454] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0eddbc17954ae989cba507455893bb in term 1.
I20260621 02:41:29.911417 14459 leader_election.cc:302] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0eddbc17954ae989cba507455893bb, a1ef4019af0548d492258bfa132feb33; no voters:
I20260621 02:41:29.911720 14726 raft_consensus.cc:2781] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:41:29.912197 14726 raft_consensus.cc:686] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 1 LEADER]: Becoming Leader. State: Replica: 0e0eddbc17954ae989cba507455893bb, State: Running, Role: LEADER
I20260621 02:41:29.912669 14726 consensus_queue.cc:237] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } }
I20260621 02:41:29.917680 14245 catalog_manager.cc:5495] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e0eddbc17954ae989cba507455893bb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0e0eddbc17954ae989cba507455893bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } health_report { overall_health: UNKNOWN } } }
I20260621 02:41:29.950676 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.945434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.0.97.2:41737)
0621 02:41:29.945627 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.945632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:29.945649 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:41:29.945787 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:29.945791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:29.945867 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:29.945990 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:41:29.946420 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.947134 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.948536 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:29.948949 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:29.949042 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:29.949600 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:41:29.950104 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:41:29.950211 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:29.950308 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 02:41:29.950372 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:41:29.958561 14390 raft_consensus.cc:1261] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Refusing update from remote peer 0e0eddbc17954ae989cba507455893bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:29.958595 14657 raft_consensus.cc:1261] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [term 1 FOLLOWER]: Refusing update from remote peer 0e0eddbc17954ae989cba507455893bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:29.959467 14731 consensus_queue.cc:1035] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:41:29.959812 14726 consensus_queue.cc:1035] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:41:29.985082 14245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:41:29.985383 14245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:41:29.989066 14245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Sending DeleteTablet for 3 replicas of tablet 3a881144963d488c8d96f5b4df89f8c4
I20260621 02:41:29.990262 14504 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a881144963d488c8d96f5b4df89f8c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33146
I20260621 02:41:29.990322 14370 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a881144963d488c8d96f5b4df89f8c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43702
I20260621 02:41:29.990510 14637 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a881144963d488c8d96f5b4df89f8c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43658
I20260621 02:41:29.991556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 14307
I20260621 02:41:30.001214 14746 tablet_replica.cc:330] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: stopping tablet replica
I20260621 02:41:30.002537 14744 tablet_replica.cc:330] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: stopping tablet replica
I20260621 02:41:30.002499 14746 raft_consensus.cc:2229] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:41:30.003238 14746 raft_consensus.cc:2258] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:30.004106 14744 raft_consensus.cc:2229] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:41:30.005220 14744 raft_consensus.cc:2258] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:30.005388 14746 ts_tablet_manager.cc:1905] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:30.007977 14744 ts_tablet_manager.cc:1905] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:30.009644 14745 tablet_replica.cc:330] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: stopping tablet replica
I20260621 02:41:30.013901 14746 ts_tablet_manager.cc:1918] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:30.014117 14746 log.cc:1196] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/3a881144963d488c8d96f5b4df89f8c4
I20260621 02:41:30.014717 14746 ts_tablet_manager.cc:1939] T 3a881144963d488c8d96f5b4df89f8c4 P a1ef4019af0548d492258bfa132feb33: Deleting consensus metadata
I20260621 02:41:30.016613 14232 catalog_manager.cc:4841] TS a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251): tablet 3a881144963d488c8d96f5b4df89f8c4 (table test-table [id=5c16f1f8d9e049f3ba3b2f90b0d04ca7]) successfully deleted
I20260621 02:41:30.017107 14744 ts_tablet_manager.cc:1918] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:30.017328 14744 log.cc:1196] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/3a881144963d488c8d96f5b4df89f8c4
I20260621 02:41:30.017719 14745 raft_consensus.cc:2229] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:41:30.017987 14744 ts_tablet_manager.cc:1939] T 3a881144963d488c8d96f5b4df89f8c4 P 0e0eddbc17954ae989cba507455893bb: Deleting consensus metadata
I20260621 02:41:30.018306 14745 raft_consensus.cc:2258] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 21 02:41:30 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:41:30.019687 14230 catalog_manager.cc:4841] TS 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737): tablet 3a881144963d488c8d96f5b4df89f8c4 (table test-table [id=5c16f1f8d9e049f3ba3b2f90b0d04ca7]) successfully deleted
I20260621 02:41:30.020162 14745 ts_tablet_manager.cc:1905] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:30.026871 14745 ts_tablet_manager.cc:1918] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:30.027031 14745 log.cc:1196] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/3a881144963d488c8d96f5b4df89f8c4
I20260621 02:41:30.027570 14745 ts_tablet_manager.cc:1939] T 3a881144963d488c8d96f5b4df89f8c4 P 83ae2e3426cb4de585d9c2297b823c8c: Deleting consensus metadata
I20260621 02:41:30.028399 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:29.993793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.0.97.1:33399)
0621 02:41:29.993957 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:29.993963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:30.008400 (+ 14437us) server_negotiation.cc:407] Connection header received
0621 02:41:30.008908 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:30.008914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:30.009034 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:30.009111 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:30.011105 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:30.012241 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:30.014375 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:30.015040 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:30.015143 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:30.020916 (+ 5773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:30.020934 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:30.020950 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:30.021001 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:30.025016 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:30.025929 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:30.025937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:30.025941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:30.026076 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:30.026635 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:30.026643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:30.026649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:30.027174 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:30.027497 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:30.027798 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 02:41:30.028029 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:41:30.029246 14430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:30.029383 14232 catalog_manager.cc:4841] TS 83ae2e3426cb4de585d9c2297b823c8c (127.0.97.1:33399): tablet 3a881144963d488c8d96f5b4df89f8c4 (table test-table [id=5c16f1f8d9e049f3ba3b2f90b0d04ca7]) successfully deleted
I20260621 02:41:30.132730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 14307
I20260621 02:41:30.150230 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 14441
Jun 21 02:41:30 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:41:30.167513 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:30.150998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.0.97.2:41737)
0621 02:41:30.151183 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:30.151192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:30.151322 (+ 130us) server_negotiation.cc:407] Connection header received
0621 02:41:30.151531 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:30.151535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:30.151607 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:30.151715 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:30.152950 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:30.153958 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:30.155894 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:30.156319 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:30.156412 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:30.161539 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:30.161554 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:30.161569 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:30.161611 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:30.164472 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:30.165233 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:30.165239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:30.165241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:30.165498 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:30.165896 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:30.165899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:30.165901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:30.166332 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:30.166579 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:30.166952 (+ 373us) server_negotiation.cc:299] Negotiation successful
0621 02:41:30.167100 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:41:30.168169 14564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:30.279805 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 14441
I20260621 02:41:30.298087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 14574
Jun 21 02:41:30 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:41:30.312933 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:30.298864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.97.3:39251)
0621 02:41:30.299044 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:30.299048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:30.299067 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:41:30.299198 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:30.299202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:30.299277 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:30.299359 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:30.300593 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:30.301426 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:30.302742 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:30.303161 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:30.303264 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:30.307703 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:30.307718 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:30.307735 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:30.307769 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:30.310617 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:30.311189 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:30.311196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:30.311198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:30.311323 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:30.311622 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:30.311625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:30.311627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:30.311965 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:30.312181 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:30.312497 (+ 316us) server_negotiation.cc:299] Negotiation successful
0621 02:41:30.312670 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:41:30.313625 14697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:30.403039 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 14574
I20260621 02:41:30.419654 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 14214
I20260621 02:41:30.420914 14276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:30.529567 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 14214
I20260621 02:41:30.550868 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34379
--webserver_interface=127.0.97.62
--webserver_port=34387
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:30.867622 14752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:30.867949 14752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:30.868110 14752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:30.875263 14752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:30.875402 14752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:30.875499 14752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:30.875592 14752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:30.875670 14752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:30.885234 14752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:30.886482 14752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:30.887725 14752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:30.910167 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:30.910243 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:30.910197 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:30.911015 14752 server_base.cc:1034] running on GCE node
I20260621 02:41:30.911979 14752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:30.913780 14752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:30.915032 14752 hybrid_clock.cc:648] HybridClock initialized: now 1782009690914953 us; error 68 us; skew 500 ppm
I20260621 02:41:30.915437 14752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:30 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:30.919214 14752 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:41:30.920969 14752 webserver.cc:460] Webserver started at http://127.0.97.62:34387/ using document root <none> and password file <none>
I20260621 02:41:30.921799 14752 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:30.921928 14752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:30.927971 14752 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:41:30.931191 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:30.932678 14752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:41:30.932842 14752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "a7285edd9e6b46c8a852e9e63e4a85a3"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "d74d8f9adaed7084d364e38557051540"
server_key_iv: "fd61f0fc268469b98c8bf4e6efa26cc5"
server_key_version: "encryptionkey@0"
I20260621 02:41:30.933976 14752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:30.951709 14752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:30.956274 14752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:30.956517 14752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:30.975847 14752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34379
I20260621 02:41:30.975853 14819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34379 every 8 connection(s)
I20260621 02:41:30.977741 14752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:41:30.981840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 14752
I20260621 02:41:30.982653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33399
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34379
--webserver_port=33269
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:41:30.985486 14820 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:41:30.988912 14820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Bootstrap starting.
I20260621 02:41:31.018471 14820 log.cc:824] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:31.033789 14820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:41:31.034451 14820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Bootstrap complete.
I20260621 02:41:31.049098 14820 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:31.050688 14820 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7285edd9e6b46c8a852e9e63e4a85a3, State: Initialized, Role: FOLLOWER
I20260621 02:41:31.051594 14820 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:31.051963 14820 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:41:31.052127 14820 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:41:31.052340 14820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:41:31.058159 14820 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:31.058866 14820 leader_election.cc:302] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7285edd9e6b46c8a852e9e63e4a85a3; no voters:
I20260621 02:41:31.059554 14820 leader_election.cc:288] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:41:31.059726 14824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:41:31.060916 14824 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 2 LEADER]: Becoming Leader. State: Replica: a7285edd9e6b46c8a852e9e63e4a85a3, State: Running, Role: LEADER
I20260621 02:41:31.061602 14824 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:31.061846 14820 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:41:31.063949 14826 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7285edd9e6b46c8a852e9e63e4a85a3. Latest consensus state: current_term: 2 leader_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } } }
I20260621 02:41:31.064327 14826 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:31.064924 14825 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } } }
I20260621 02:41:31.065250 14825 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:31.067442 14830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:41:31.076333 14830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:41:31.078013 14830 catalog_manager.cc:1259] Loaded cluster ID: 8b901cf13c0044b0816dd3db90914fdf
I20260621 02:41:31.078156 14830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:41:31.081241 14830 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 02:41:31.099387 14830 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 02:41:31.102484 14842 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:41:31.108596 14830 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:41:31.110265 14830 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Loaded TSK: 0
I20260621 02:41:31.111724 14830 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:41:31.346915 14822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:31.347225 14822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:31.347386 14822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:31.356791 14822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:31.356957 14822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:31.357172 14822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:41:31.369544 14822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33269
--tserver_master_addrs=127.0.97.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:31.370827 14822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:31.372151 14822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:31.386935 14848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:31.387118 14849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:31.387137 14851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:31.387938 14822 server_base.cc:1034] running on GCE node
I20260621 02:41:31.388545 14822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:31.389506 14822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:31.390769 14822 hybrid_clock.cc:648] HybridClock initialized: now 1782009691390703 us; error 49 us; skew 500 ppm
I20260621 02:41:31.391111 14822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:31 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:31.394368 14822 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:41:31.395838 14822 webserver.cc:460] Webserver started at http://127.0.97.1:33269/ using document root <none> and password file <none>
I20260621 02:41:31.396510 14822 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:31.396667 14822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:31.401998 14822 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:41:31.404839 14858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:31.406344 14822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:41:31.406530 14822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "83ae2e3426cb4de585d9c2297b823c8c"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "ef3aad1a257124edd81461a0f17d711e"
server_key_iv: "5aa320b60498e5b2e67a9e50a4cc5b18"
server_key_version: "encryptionkey@0"
I20260621 02:41:31.407521 14822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:31.421452 14822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:31.425508 14822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:31.425777 14822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:31.426714 14822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:31.428133 14822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:31.428264 14822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:31.428391 14822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:31.428478 14822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:31.461366 14822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33399
I20260621 02:41:31.461474 14970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33399 every 8 connection(s)
I20260621 02:41:31.462857 14822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:41:31.469957 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 14822
I20260621 02:41:31.470582 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41737
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34379
--webserver_port=33513
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:41:31 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:31.490697 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:31.465849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38905 (local address 127.0.97.62:34379)
0621 02:41:31.466562 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:31.466577 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:31.466934 (+ 357us) server_negotiation.cc:407] Connection header received
0621 02:41:31.469049 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:31.469061 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:31.469442 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:31.469808 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:31.471396 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:31.472598 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:31.474201 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:31.474843 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:31.475037 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:31.481921 (+ 6884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:31.481934 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:31.481955 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:31.482005 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:31.486227 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:31.487011 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:31.487018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:31.487023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:31.487190 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:31.487636 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:31.487651 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:31.487654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:31.488096 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:31.488326 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:31.489062 (+ 736us) server_negotiation.cc:299] Negotiation successful
0621 02:41:31.489480 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":284,"threads_started":1}
I20260621 02:41:31.490710 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:31.465888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:38905)
0621 02:41:31.466762 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:31.466801 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:31.468488 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:31.469927 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:31.469940 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:31.470556 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:31.471152 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:31.471170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:31.472768 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:31.472781 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:31.474047 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:31.474059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:31.475080 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:31.475088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:31.475361 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:31.479180 (+ 3819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:31.479203 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:31.481615 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:31.486396 (+ 4781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:31.486402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:31.486413 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:31.486838 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:31.487313 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:31.487318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:31.487322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:31.487507 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:31.488256 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:31.488265 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:31.488577 (+ 312us) client_negotiation.cc:769] Sending connection context
0621 02:41:31.488846 (+ 269us) client_negotiation.cc:240] Negotiation successful
0621 02:41:31.489400 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":379,"threads_started":1}
I20260621 02:41:31.493963 14971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34379
I20260621 02:41:31.494285 14971 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:31.495079 14971 heartbeater.cc:511] Master 127.0.97.62:34379 requested a full tablet report, sending...
I20260621 02:41:31.497543 14785 ts_manager.cc:194] Registered new tserver with Master: 83ae2e3426cb4de585d9c2297b823c8c (127.0.97.1:33399)
I20260621 02:41:31.499577 14785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38905
W20260621 02:41:31.797242 14975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:31.797520 14975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:31.797612 14975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:31.804674 14975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:31.804776 14975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:31.804883 14975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:41:31.814554 14975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33513
--tserver_master_addrs=127.0.97.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:31.815719 14975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:31.816954 14975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:31.829602 14981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:31.829669 14983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:31.830065 14980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:31.830765 14975 server_base.cc:1034] running on GCE node
I20260621 02:41:31.831343 14975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:31.832223 14975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:31.833387 14975 hybrid_clock.cc:648] HybridClock initialized: now 1782009691833339 us; error 30 us; skew 500 ppm
I20260621 02:41:31.833688 14975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:31 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:31.836622 14975 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:41:31.838119 14975 webserver.cc:460] Webserver started at http://127.0.97.2:33513/ using document root <none> and password file <none>
I20260621 02:41:31.838819 14975 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:31.838929 14975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:31.843966 14975 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:41:31.846498 14990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:31.847857 14975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:41:31.847994 14975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "0e0eddbc17954ae989cba507455893bb"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "fdf55df3496b8b4567427455897201fa"
server_key_iv: "f15ea3595df8f8d474f18c1d63dc436d"
server_key_version: "encryptionkey@0"
I20260621 02:41:31.848886 14975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:31.874318 14975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:31.878400 14975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:31.878628 14975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:31.879576 14975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:31.880812 14975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:31.880898 14975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:31.880967 14975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:31.881021 14975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:31.914273 14975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41737
I20260621 02:41:31.914366 15102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41737 every 8 connection(s)
I20260621 02:41:31.915692 14975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:41:31.924458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 14975
I20260621 02:41:31.925209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39251
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34379
--webserver_port=38093
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:41:31 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:31.937196 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:31.918238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48853 (local address 127.0.97.62:34379)
0621 02:41:31.918447 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:31.918453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:31.919612 (+ 1159us) server_negotiation.cc:407] Connection header received
0621 02:41:31.921493 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:31.921498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:31.921568 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:31.921657 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:31.923314 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:31.924361 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:31.926840 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:31.927263 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:31.927359 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:31.931543 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:31.931548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:31.931552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:31.931584 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:31.934242 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:31.934937 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:31.934949 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:31.934952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:31.935124 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:31.935551 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:31.935556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:31.935559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:31.935816 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:31.935971 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:31.936614 (+ 643us) server_negotiation.cc:299] Negotiation successful
0621 02:41:31.936838 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:41:31.938215 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:31.918743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:48853)
0621 02:41:31.919448 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:31.919491 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:31.921128 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:31.921867 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:31.921874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:31.922433 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:31.923105 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:31.923131 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:31.924534 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:31.924542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:31.926675 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:31.926694 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:31.927493 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:31.927503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:31.927832 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:31.928580 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:31.928614 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:31.931327 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:31.934385 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:31.934394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:31.934409 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:31.934812 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:31.935244 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:31.935249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:31.935252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:31.935435 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:31.935945 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:31.935953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:31.936260 (+ 307us) client_negotiation.cc:769] Sending connection context
0621 02:41:31.936511 (+ 251us) client_negotiation.cc:240] Negotiation successful
0621 02:41:31.937003 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":287,"threads_started":1}
I20260621 02:41:31.939635 15103 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34379
I20260621 02:41:31.939934 15103 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:31.940611 15103 heartbeater.cc:511] Master 127.0.97.62:34379 requested a full tablet report, sending...
I20260621 02:41:31.942126 14785 ts_manager.cc:194] Registered new tserver with Master: 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737)
I20260621 02:41:31.943529 14785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48853
W20260621 02:41:32.250229 15106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:32.250519 15106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:32.250614 15106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:32.258189 15106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:32.258311 15106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:32.258427 15106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:41:32.268483 15106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38093
--tserver_master_addrs=127.0.97.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:32.269824 15106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:32.271059 15106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:32.284399 15111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:32.284260 15112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:32.284296 15114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:32.285751 15106 server_base.cc:1034] running on GCE node
I20260621 02:41:32.286422 15106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:32.287422 15106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:32.288662 15106 hybrid_clock.cc:648] HybridClock initialized: now 1782009692288608 us; error 45 us; skew 500 ppm
I20260621 02:41:32.289002 15106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:32 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:32.292318 15106 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:41:32.293995 15106 webserver.cc:460] Webserver started at http://127.0.97.3:38093/ using document root <none> and password file <none>
I20260621 02:41:32.294724 15106 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:32.294888 15106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:32.300426 15106 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:41:32.303277 15121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:32.304607 15106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:41:32.304804 15106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "a1ef4019af0548d492258bfa132feb33"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "86e3ba938e2060463b5909199e94a17b"
server_key_iv: "2493b1d2a48914760e3698d99bac4efb"
server_key_version: "encryptionkey@0"
I20260621 02:41:32.305856 15106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:32.328132 15106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:32.332165 15106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:32.332437 15106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:32.333464 15106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:32.334769 15106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:32.334899 15106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:32.335036 15106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:32.335134 15106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:32.370599 15106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39251
I20260621 02:41:32.370685 15233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39251 every 8 connection(s)
I20260621 02:41:32.372030 15106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:41:32.379616 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 15106
Jun 21 02:41:32 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:32.393663 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.374584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39793 (local address 127.0.97.62:34379)
0621 02:41:32.374801 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.374807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.376041 (+ 1234us) server_negotiation.cc:407] Connection header received
0621 02:41:32.377939 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.377944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.378023 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.378116 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:32.379564 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.380586 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.382766 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.383177 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.383326 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.387003 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:32.387008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:32.387011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:32.387040 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:32.390003 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.390991 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.390997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.391002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.391157 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.391717 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.391721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.391723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.392000 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:32.392110 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.393087 (+ 977us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.393335 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:41:32.394750 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.375109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:39793)
0621 02:41:32.375885 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.375922 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:32.377661 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:32.378333 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:32.378341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:32.378845 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:32.379375 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.379393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.380746 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.380792 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.382590 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.382602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.383426 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.383434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.383736 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.384486 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:32.384520 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:32.386786 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:32.390209 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:32.390222 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:32.390244 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:32.390834 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:32.391318 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:32.391324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:32.391328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:32.391589 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:32.392137 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:32.392147 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:32.392610 (+ 463us) client_negotiation.cc:769] Sending connection context
0621 02:41:32.392917 (+ 307us) client_negotiation.cc:240] Negotiation successful
0621 02:41:32.393369 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":273,"threads_started":1}
I20260621 02:41:32.396030 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.380638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.0.97.62:34379)
0621 02:41:32.381521 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.381527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.381840 (+ 313us) server_negotiation.cc:407] Connection header received
0621 02:41:32.381978 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.381985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.382064 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.382174 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:32.383651 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.384696 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.386066 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.386568 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.386696 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.389251 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:32.389259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:32.389262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:32.389356 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:32.392639 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.393522 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.393527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.393530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.393671 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.394188 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.394193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.394196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.394518 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:32.394775 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.395327 (+ 552us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.395743 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":582,"threads_started":1}
I20260621 02:41:32.396319 15234 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34379
I20260621 02:41:32.396646 15234 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:32.397481 15234 heartbeater.cc:511] Master 127.0.97.62:34379 requested a full tablet report, sending...
I20260621 02:41:32.398948 14785 ts_manager.cc:194] Registered new tserver with Master: a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251)
I20260621 02:41:32.399731 14785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39793
I20260621 02:41:32.410876 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:41:32.417430 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.411930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.0.97.62:34379)
0621 02:41:32.412110 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.412114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.412490 (+ 376us) server_negotiation.cc:407] Connection header received
0621 02:41:32.412663 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.412666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.412811 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.412932 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:41:32.413356 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.414046 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.415387 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.415780 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.415868 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.416298 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:41:32.416797 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:41:32.416901 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.417044 (+ 143us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.417105 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:41:32.418349 14785 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:41:32.425611 14785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:41:32.459996 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.437730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.0.97.2:41737)
0621 02:41:32.447664 (+ 9934us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.447679 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.448625 (+ 946us) server_negotiation.cc:407] Connection header received
0621 02:41:32.448979 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.448991 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.449392 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.449604 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:32.450887 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.453058 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.456076 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.457212 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.458668 (+ 1456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.458897 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.459491 (+ 594us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.459680 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9709,"thread_start_us":9972,"threads_started":1}
I20260621 02:41:32.460019 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.437204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41737 (local address 127.0.0.1:40936)
0621 02:41:32.448483 (+ 11279us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.448508 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:32.448755 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:32.450016 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:32.450028 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:32.450070 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:32.450607 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.450622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.453389 (+ 2767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.453398 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.455908 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.455922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.459040 (+ 3118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.459047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.459226 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.459279 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 02:41:32.459415 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 02:41:32.459599 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11110,"mutex_wait_us":190,"thread_start_us":10721,"threads_started":1}
I20260621 02:41:32.461508 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.438408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39251 (local address 127.0.0.1:58162)
0621 02:41:32.453246 (+ 14838us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.453265 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:32.453395 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:32.454041 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:32.454046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:32.454073 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:32.454434 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.454445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.456313 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.456319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.458493 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.458513 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.460558 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.460564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.460749 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.460772 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:41:32.461002 (+ 230us) client_negotiation.cc:240] Negotiation successful
0621 02:41:32.461117 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14729,"mutex_wait_us":54,"thread_start_us":14612,"threads_started":1}
I20260621 02:41:32.461946 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.439069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.0.97.3:39251)
0621 02:41:32.453448 (+ 14379us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.453458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.453491 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:41:32.453592 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.453602 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.453830 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.454019 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:32.454680 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.456181 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.458670 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.460058 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.460193 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.460427 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.461382 (+ 955us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.461595 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14154,"thread_start_us":13570,"threads_started":1}
I20260621 02:41:32.464922 15038 tablet_service.cc:1467] Processing CreateTablet for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab (DEFAULT_TABLE table=test-table [id=38ea445a77f440d5995f5717e2c63856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:32.465359 15169 tablet_service.cc:1467] Processing CreateTablet for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab (DEFAULT_TABLE table=test-table [id=38ea445a77f440d5995f5717e2c63856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:32.466239 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.437204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:54272)
0621 02:41:32.438129 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.438187 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:32.438403 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:32.454429 (+ 16026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:32.454434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:32.454463 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:32.454963 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.454974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.456694 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.456704 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.458813 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.458823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.465459 (+ 6636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.465465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.465640 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.465683 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 02:41:32.465751 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 02:41:32.465875 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":20,"thread_start_us":446,"threads_started":1}
I20260621 02:41:32.466409 15038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:32.466584 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:32.467257 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.437419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.0.97.1:33399)
0621 02:41:32.453889 (+ 16470us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.453897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.453927 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:41:32.454016 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.454023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.454227 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.454431 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:32.455168 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.456549 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.458956 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.460377 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.466448 (+ 6071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.466671 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.466738 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.466899 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16313,"thread_start_us":16203,"threads_started":1}
I20260621 02:41:32.471028 14906 tablet_service.cc:1467] Processing CreateTablet for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab (DEFAULT_TABLE table=test-table [id=38ea445a77f440d5995f5717e2c63856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:32.472072 14906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:32.481798 15250 tablet_bootstrap.cc:492] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: Bootstrap starting.
I20260621 02:41:32.484017 15251 tablet_bootstrap.cc:492] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: Bootstrap starting.
I20260621 02:41:32.486255 15250 tablet_bootstrap.cc:654] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:32.486784 15251 tablet_bootstrap.cc:654] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:32.487914 15251 log.cc:824] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:32.488705 15250 log.cc:824] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:32.489329 15252 tablet_bootstrap.cc:492] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Bootstrap starting.
I20260621 02:41:32.491132 15251 tablet_bootstrap.cc:492] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: No bootstrap required, opened a new log
I20260621 02:41:32.491405 15251 ts_tablet_manager.cc:1397] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260621 02:41:32.492657 15252 tablet_bootstrap.cc:654] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:32.493849 15252 log.cc:824] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:32.496882 15252 tablet_bootstrap.cc:492] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: No bootstrap required, opened a new log
I20260621 02:41:32.496882 15250 tablet_bootstrap.cc:492] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: No bootstrap required, opened a new log
I20260621 02:41:32.497164 15252 ts_tablet_manager.cc:1397] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260621 02:41:32.497150 15250 ts_tablet_manager.cc:1397] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20260621 02:41:32.499678 15251 raft_consensus.cc:348] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:32.500291 15251 raft_consensus.cc:374] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:32.500435 15251 raft_consensus.cc:729] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ef4019af0548d492258bfa132feb33, State: Initialized, Role: FOLLOWER
I20260621 02:41:32.501185 15251 consensus_queue.cc:260] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:32.502367 15234 heartbeater.cc:503] Master 127.0.97.62:34379 was elected leader, sending a full tablet report...
I20260621 02:41:32.503000 14971 heartbeater.cc:503] Master 127.0.97.62:34379 was elected leader, sending a full tablet report...
I20260621 02:41:32.507427 15251 ts_tablet_manager.cc:1428] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260621 02:41:32.507907 15250 raft_consensus.cc:348] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:32.508388 15250 raft_consensus.cc:374] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:32.508520 15250 raft_consensus.cc:729] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0eddbc17954ae989cba507455893bb, State: Initialized, Role: FOLLOWER
I20260621 02:41:32.508980 15252 raft_consensus.cc:348] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:32.509173 15250 consensus_queue.cc:260] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:32.509438 15252 raft_consensus.cc:374] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:32.509546 15252 raft_consensus.cc:729] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ae2e3426cb4de585d9c2297b823c8c, State: Initialized, Role: FOLLOWER
I20260621 02:41:32.510241 15252 consensus_queue.cc:260] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:32.510545 15103 heartbeater.cc:503] Master 127.0.97.62:34379 was elected leader, sending a full tablet report...
I20260621 02:41:32.511121 15250 ts_tablet_manager.cc:1428] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260621 02:41:32.512030 15252 ts_tablet_manager.cc:1428] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260621 02:41:32.540534 15257 raft_consensus.cc:482] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:32.540802 15257 raft_consensus.cc:504] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:32.542443 15257 leader_election.cc:288] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ae2e3426cb4de585d9c2297b823c8c (127.0.97.1:33399), a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251)
I20260621 02:41:32.549629 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.542891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36553 (local address 127.0.97.1:33399)
0621 02:41:32.543219 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.543228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.543518 (+ 290us) server_negotiation.cc:407] Connection header received
0621 02:41:32.543848 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.543854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.543938 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.544034 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:32.544691 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.545738 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.547647 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.548548 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.548657 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.548733 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.549214 (+ 481us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.549380 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 02:41:32.549724 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.543647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34093 (local address 127.0.97.3:39251)
0621 02:41:32.543860 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.543866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.544322 (+ 456us) server_negotiation.cc:407] Connection header received
0621 02:41:32.544447 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.544455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.544537 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.544636 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:32.545251 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.546118 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.547939 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.548781 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.548901 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.549002 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.549349 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.549483 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:41:32.549872 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.543436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39251 (local address 127.0.97.2:34093)
0621 02:41:32.544209 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.544225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:32.544337 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:32.544676 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:32.544680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:32.544704 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:32.545083 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.545093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.546262 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.546266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.547786 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.547800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.548977 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.548982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.549176 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.549207 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:41:32.549325 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 02:41:32.549456 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":24,"thread_start_us":465,"threads_started":1}
I20260621 02:41:32.549909 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.542700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.97.2:36553)
0621 02:41:32.543414 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.543434 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:32.543633 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:32.544077 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:32.544084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:32.544110 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:32.544532 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.544546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.545868 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.545873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.547488 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:32.547499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.548772 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.548778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:32.548967 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.549069 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 02:41:32.549147 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 02:41:32.549245 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":73,"thread_start_us":372,"threads_started":1}
I20260621 02:41:32.550931 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b422fdbcb34f2f8fa07616c3f5d7ab" candidate_uuid: "0e0eddbc17954ae989cba507455893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ef4019af0548d492258bfa132feb33" is_pre_election: true
I20260621 02:41:32.550917 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b422fdbcb34f2f8fa07616c3f5d7ab" candidate_uuid: "0e0eddbc17954ae989cba507455893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ae2e3426cb4de585d9c2297b823c8c" is_pre_election: true
I20260621 02:41:32.551396 14926 raft_consensus.cc:2454] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0eddbc17954ae989cba507455893bb in term 0.
I20260621 02:41:32.551402 15189 raft_consensus.cc:2454] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0eddbc17954ae989cba507455893bb in term 0.
I20260621 02:41:32.552060 14993 leader_election.cc:302] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0eddbc17954ae989cba507455893bb, a1ef4019af0548d492258bfa132feb33; no voters:
I20260621 02:41:32.552495 15257 raft_consensus.cc:2781] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:41:32.552650 15257 raft_consensus.cc:482] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:41:32.552771 15257 raft_consensus.cc:3037] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:32.555002 15257 raft_consensus.cc:504] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:32.555581 15257 leader_election.cc:288] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [CANDIDATE]: Term 1 election: Requested vote from peers 83ae2e3426cb4de585d9c2297b823c8c (127.0.97.1:33399), a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251)
I20260621 02:41:32.556051 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b422fdbcb34f2f8fa07616c3f5d7ab" candidate_uuid: "0e0eddbc17954ae989cba507455893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ae2e3426cb4de585d9c2297b823c8c"
I20260621 02:41:32.556107 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b422fdbcb34f2f8fa07616c3f5d7ab" candidate_uuid: "0e0eddbc17954ae989cba507455893bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ef4019af0548d492258bfa132feb33"
I20260621 02:41:32.556238 14926 raft_consensus.cc:3037] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:32.556331 15189 raft_consensus.cc:3037] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:32.558531 15189 raft_consensus.cc:2454] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0eddbc17954ae989cba507455893bb in term 1.
I20260621 02:41:32.558543 14926 raft_consensus.cc:2454] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0eddbc17954ae989cba507455893bb in term 1.
I20260621 02:41:32.558964 14993 leader_election.cc:302] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0eddbc17954ae989cba507455893bb, 83ae2e3426cb4de585d9c2297b823c8c; no voters:
I20260621 02:41:32.559280 15257 raft_consensus.cc:2781] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:41:32.559742 15257 raft_consensus.cc:686] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 1 LEADER]: Becoming Leader. State: Replica: 0e0eddbc17954ae989cba507455893bb, State: Running, Role: LEADER
I20260621 02:41:32.560326 15257 consensus_queue.cc:237] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:32.564816 14785 catalog_manager.cc:5495] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e0eddbc17954ae989cba507455893bb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0e0eddbc17954ae989cba507455893bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } health_report { overall_health: UNKNOWN } } }
I20260621 02:41:32.602646 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.597457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.0.97.2:41737)
0621 02:41:32.597664 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.597669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.597688 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:41:32.597840 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.597844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.597930 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.598054 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:41:32.598435 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.599153 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.600506 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.600905 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.600994 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.601515 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:41:32.602033 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:41:32.602141 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.602242 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.602307 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:41:32.611039 14926 raft_consensus.cc:1261] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Refusing update from remote peer 0e0eddbc17954ae989cba507455893bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:32.611063 15189 raft_consensus.cc:1261] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [term 1 FOLLOWER]: Refusing update from remote peer 0e0eddbc17954ae989cba507455893bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:32.611987 15261 consensus_queue.cc:1035] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:41:32.612411 15257 consensus_queue.cc:1035] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:41:32.623514 15235 tablet.cc:2369] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:41:32.625869 15263 mvcc.cc:205] Tried to move back new op lower bound from 7299111700923940864 to 7299111700731473920. Current Snapshot: MvccSnapshot[applied={T|T < 7299111700923940864}]
I20260621 02:41:32.627560 15265 mvcc.cc:205] Tried to move back new op lower bound from 7299111700923940864 to 7299111700731473920. Current Snapshot: MvccSnapshot[applied={T|T < 7299111700923940864}]
I20260621 02:41:32.640053 14784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:41:32.640355 14784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:41:32.644130 14784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Sending DeleteTablet for 3 replicas of tablet 51b422fdbcb34f2f8fa07616c3f5d7ab
I20260621 02:41:32.645522 15038 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40936
I20260621 02:41:32.645535 14906 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54272
I20260621 02:41:32.645788 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 14822
I20260621 02:41:32.646631 15169 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58162
I20260621 02:41:32.660583 15273 tablet_replica.cc:330] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: stopping tablet replica
I20260621 02:41:32.661310 15273 raft_consensus.cc:2229] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:41:32.662242 15273 raft_consensus.cc:2258] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:32.665120 15273 ts_tablet_manager.cc:1905] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:32.668016 15274 tablet_replica.cc:330] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: stopping tablet replica
I20260621 02:41:32.668526 15274 raft_consensus.cc:2229] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:41:32.669337 15274 raft_consensus.cc:2258] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:32.669454 15275 tablet_replica.cc:330] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: stopping tablet replica
I20260621 02:41:32.669869 15275 raft_consensus.cc:2229] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:41:32.670346 15275 raft_consensus.cc:2258] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:32.670604 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.647006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.0.97.1:33399)
0621 02:41:32.647220 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.647226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.647248 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:41:32.647322 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.647330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.647444 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.647565 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:32.650119 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.651669 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.654410 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.655336 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.655486 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.661503 (+ 6017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:32.661528 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:32.661553 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:32.661611 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:32.666800 (+ 5189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.667994 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.668012 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.668018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.668222 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.668792 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.668800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.668806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.669430 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:32.669731 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.670037 (+ 306us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.670260 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 02:41:32.671307 14966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:32.673221 15275 ts_tablet_manager.cc:1905] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:32.677008 15273 ts_tablet_manager.cc:1918] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:32.677196 15273 log.cc:1196] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/51b422fdbcb34f2f8fa07616c3f5d7ab
I20260621 02:41:32.677809 15273 ts_tablet_manager.cc:1939] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 0e0eddbc17954ae989cba507455893bb: Deleting consensus metadata
I20260621 02:41:32.679632 14769 catalog_manager.cc:4841] TS 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737): tablet 51b422fdbcb34f2f8fa07616c3f5d7ab (table test-table [id=38ea445a77f440d5995f5717e2c63856]) successfully deleted
I20260621 02:41:32.683897 15275 ts_tablet_manager.cc:1918] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:32.684078 15275 log.cc:1196] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/51b422fdbcb34f2f8fa07616c3f5d7ab
I20260621 02:41:32.684522 15275 ts_tablet_manager.cc:1939] T 51b422fdbcb34f2f8fa07616c3f5d7ab P a1ef4019af0548d492258bfa132feb33: Deleting consensus metadata
I20260621 02:41:32.686122 14771 catalog_manager.cc:4841] TS a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251): tablet 51b422fdbcb34f2f8fa07616c3f5d7ab (table test-table [id=38ea445a77f440d5995f5717e2c63856]) successfully deleted
I20260621 02:41:32.791324 15274 ts_tablet_manager.cc:1905] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:32.793412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 14822
I20260621 02:41:32.815693 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 14975
W20260621 02:41:32.815719 14771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33399 (error 108)
I20260621 02:41:32.818954 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.818442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:54298)
0621 02:41:32.818631 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.818747 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260621 02:41:32.819274 14771 catalog_manager.cc:4584] TS 83ae2e3426cb4de585d9c2297b823c8c (127.0.97.1:33399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51b422fdbcb34f2f8fa07616c3f5d7ab: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
I20260621 02:41:32.830673 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.816754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.0.97.2:41737)
0621 02:41:32.816951 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.816957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.816978 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:41:32.817125 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.817131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.817212 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.817345 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:32.819093 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.820038 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.821448 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.821898 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.822046 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.824679 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:32.824702 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:32.824720 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:32.824760 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:32.827747 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.828470 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.828482 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.828488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.828623 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.829008 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.829014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.829017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.829459 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:32.829787 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.830105 (+ 318us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.830310 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:41:32.831359 15098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:32.873008 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.872445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:54300)
0621 02:41:32.872683 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.872795 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260621 02:41:32.874203 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.873823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:54312)
0621 02:41:32.873980 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.874075 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 02:41:32.934736 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.934178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:54328)
0621 02:41:32.934408 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.934523 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 02:41:32.935801 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.935441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:54344)
0621 02:41:32.935593 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:32.935676 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260621 02:41:32.945451 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 14975
I20260621 02:41:32.964772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 15106
I20260621 02:41:32.984951 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:32.969995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.0.97.3:39251)
0621 02:41:32.970232 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:32.970237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:32.971470 (+ 1233us) server_negotiation.cc:407] Connection header received
0621 02:41:32.971616 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:32.971621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:32.971693 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:32.971795 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:32.973066 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.973881 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.975493 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:32.975937 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:32.976031 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:32.979393 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:32.979408 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:32.979425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:32.979456 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:32.982256 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.983087 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.983094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.983096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.983248 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:32.983623 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:32.983626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:32.983628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:32.983951 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:32.984157 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:32.984498 (+ 341us) server_negotiation.cc:299] Negotiation successful
0621 02:41:32.984664 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:41:32.985659 15229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:33.016587 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:33.016068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:54358)
0621 02:41:33.016307 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:33.016416 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260621 02:41:33.017792 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:33.017401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:54366)
0621 02:41:33.017600 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:33.017677 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260621 02:41:33.075186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 15106
I20260621 02:41:33.091807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 14752
I20260621 02:41:33.092911 14815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:33.212898 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:33.212319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33399 (local address 127.0.0.1:54378)
0621 02:41:33.212518 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:33.212635 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260621 02:41:33.213097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 14752
I20260621 02:41:33.234777 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34379
--webserver_interface=127.0.97.62
--webserver_port=34387
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:33.558192 15281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:33.558470 15281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:33.558648 15281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:33.566176 15281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:33.566274 15281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:33.566372 15281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:33.566478 15281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:33.566538 15281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:33.576208 15281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:33.577530 15281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:33.578811 15281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:33.590303 15286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:33.590334 15289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:33.590250 15287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:33.591013 15281 server_base.cc:1034] running on GCE node
I20260621 02:41:33.591727 15281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:33.593032 15281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:33.594276 15281 hybrid_clock.cc:648] HybridClock initialized: now 1782009693594194 us; error 75 us; skew 500 ppm
I20260621 02:41:33.594633 15281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:33 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:33.597681 15281 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:41:33.599114 15281 webserver.cc:460] Webserver started at http://127.0.97.62:34387/ using document root <none> and password file <none>
I20260621 02:41:33.599774 15281 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:33.599920 15281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:33.605386 15281 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:41:33.608209 15296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:33.609671 15281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:41:33.609854 15281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "a7285edd9e6b46c8a852e9e63e4a85a3"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "d74d8f9adaed7084d364e38557051540"
server_key_iv: "fd61f0fc268469b98c8bf4e6efa26cc5"
server_key_version: "encryptionkey@0"
I20260621 02:41:33.610841 15281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:33.623651 15281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:33.627777 15281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:33.628082 15281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:33.646459 15281 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34379
I20260621 02:41:33.646502 15347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34379 every 8 connection(s)
I20260621 02:41:33.647866 15281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:41:33.654558 15348 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:41:33.655213 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 15281
I20260621 02:41:33.655983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33399
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34379
--webserver_port=33269
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:41:33.657877 15348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Bootstrap starting.
I20260621 02:41:33.677838 15348 log.cc:824] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:33.693748 15348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:41:33.694278 15348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3: Bootstrap complete.
I20260621 02:41:33.703238 15348 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:33.704078 15348 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7285edd9e6b46c8a852e9e63e4a85a3, State: Initialized, Role: FOLLOWER
I20260621 02:41:33.704722 15348 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:33.704959 15348 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:41:33.705077 15348 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:41:33.705317 15348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 2 FOLLOWER]: Advancing to term 3
I20260621 02:41:33.709118 15348 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:33.709887 15348 leader_election.cc:302] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7285edd9e6b46c8a852e9e63e4a85a3; no voters:
I20260621 02:41:33.710662 15348 leader_election.cc:288] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 02:41:33.710795 15352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 02:41:33.711999 15352 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [term 3 LEADER]: Becoming Leader. State: Replica: a7285edd9e6b46c8a852e9e63e4a85a3, State: Running, Role: LEADER
I20260621 02:41:33.712625 15352 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } }
I20260621 02:41:33.712934 15348 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:41:33.715305 15354 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7285edd9e6b46c8a852e9e63e4a85a3. Latest consensus state: current_term: 3 leader_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } } }
I20260621 02:41:33.715643 15354 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:33.715890 15353 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7285edd9e6b46c8a852e9e63e4a85a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34379 } } }
I20260621 02:41:33.716116 15353 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7285edd9e6b46c8a852e9e63e4a85a3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:33.727571 15358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:41:33.736680 15358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:41:33.738168 15358 catalog_manager.cc:1259] Loaded cluster ID: 8b901cf13c0044b0816dd3db90914fdf
I20260621 02:41:33.738356 15358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 02:41:33.740761 15358 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff60fc4d9ad google::LogMessage::Fail() at ??:0
@ 0x7ff60fc5088a google::LogMessage::SendToLog() at ??:0
@ 0x7ff60fc4d4de google::LogMessage::Flush() at ??:0
@ 0x7ff60fc4ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff625680512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff62567d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff625705610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff625705439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff612899345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff61289c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff61289c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff612877eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff615b516db start_thread at ??:0
@ 0x7ff60eae761f clone at ??:0
W20260621 02:41:34.026087 15350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:34.026443 15350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:34.026613 15350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:34.034305 15350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:34.034468 15350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:34.034703 15350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:41:34.045527 15350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33269
--tserver_master_addrs=127.0.97.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:34.046916 15350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:34.048270 15350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:34.061821 15390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:34.062258 15389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:34.062312 15392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:34.063000 15350 server_base.cc:1034] running on GCE node
I20260621 02:41:34.063634 15350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:34.064606 15350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:34.065807 15350 hybrid_clock.cc:648] HybridClock initialized: now 1782009694065763 us; error 34 us; skew 500 ppm
I20260621 02:41:34.066099 15350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:34 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:34.069267 15350 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:41:34.070802 15350 webserver.cc:460] Webserver started at http://127.0.97.1:33269/ using document root <none> and password file <none>
I20260621 02:41:34.071492 15350 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:34.071627 15350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:34.077239 15350 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.002s
I20260621 02:41:34.080277 15399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:34.081746 15350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 02:41:34.081923 15350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "83ae2e3426cb4de585d9c2297b823c8c"
format_stamp: "Formatted at 2026-06-21 02:41:28 on dist-test-slave-jkn7"
server_key: "ef3aad1a257124edd81461a0f17d711e"
server_key_iv: "5aa320b60498e5b2e67a9e50a4cc5b18"
server_key_version: "encryptionkey@0"
I20260621 02:41:34.082899 15350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:34.130079 15350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:34.134119 15350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:34.134414 15350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:34.135560 15350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:34.137320 15406 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260621 02:41:34.142931 15350 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 02:41:34.143074 15350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260621 02:41:34.143213 15350 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 02:41:34.146333 15350 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 02:41:34.146406 15350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260621 02:41:34.146790 15406 tablet_bootstrap.cc:492] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Bootstrap starting.
I20260621 02:41:34.155422 15406 log.cc:824] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:34.171661 15406 tablet_bootstrap.cc:492] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:41:34.172676 15406 tablet_bootstrap.cc:492] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Bootstrap complete.
I20260621 02:41:34.173417 15406 ts_tablet_manager.cc:1397] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.007s
I20260621 02:41:34.182614 15406 raft_consensus.cc:348] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:34.184320 15406 raft_consensus.cc:729] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ae2e3426cb4de585d9c2297b823c8c, State: Initialized, Role: FOLLOWER
I20260621 02:41:34.185411 15406 consensus_queue.cc:260] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:34.187315 15406 ts_tablet_manager.cc:1428] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260621 02:41:34.192777 15350 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33399
I20260621 02:41:34.192819 15514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33399 every 8 connection(s)
I20260621 02:41:34.194165 15350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:41:34.198832 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:34.196883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:55109)
0621 02:41:34.197792 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:34.197985 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":457,"threads_started":1}
W20260621 02:41:34.199589 15515 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34379 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34379: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
I20260621 02:41:34.199637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 15350
I20260621 02:41:34.200357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41737
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34379
--webserver_port=33513
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:41:34.200510 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:34.200078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:40875)
0621 02:41:34.200274 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:34.200365 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 02:41:34.201416 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:34.201034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:44505)
0621 02:41:34.201173 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:34.201240 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:41:34.201697 15515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:41:34.519696 15518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:34.519989 15518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:34.520154 15518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:34.527515 15518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:34.527635 15518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:34.527813 15518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:41:34.537736 15518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33513
--tserver_master_addrs=127.0.97.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:34.539012 15518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:34.540282 15518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:34.553469 15523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:34.553442 15524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:34.553506 15526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:34.554303 15518 server_base.cc:1034] running on GCE node
I20260621 02:41:34.554889 15518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:34.555804 15518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:34.557063 15518 hybrid_clock.cc:648] HybridClock initialized: now 1782009694557015 us; error 56 us; skew 500 ppm
I20260621 02:41:34.557468 15518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:34 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:34.560509 15518 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:41:34.562011 15518 webserver.cc:460] Webserver started at http://127.0.97.2:33513/ using document root <none> and password file <none>
I20260621 02:41:34.562628 15518 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:34.562731 15518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:34.567787 15518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260621 02:41:34.572104 15533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:34.573472 15518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:41:34.573607 15518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "0e0eddbc17954ae989cba507455893bb"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "fdf55df3496b8b4567427455897201fa"
server_key_iv: "f15ea3595df8f8d474f18c1d63dc436d"
server_key_version: "encryptionkey@0"
I20260621 02:41:34.574525 15518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:34.626147 15518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:34.630111 15518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:34.630333 15518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:34.631239 15518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:34.632555 15518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:34.632647 15518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:34.632722 15518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:34.632777 15518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:34.666014 15518 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41737
I20260621 02:41:34.666101 15645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41737 every 8 connection(s)
I20260621 02:41:34.667421 15518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:41:34.672047 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:34.670280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:50177)
0621 02:41:34.671008 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:34.671248 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":284,"threads_started":1}
W20260621 02:41:34.672803 15646 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34379 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34379: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
I20260621 02:41:34.673026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 15518
I20260621 02:41:34.673660 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:34.673354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:55493)
0621 02:41:34.673497 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:34.673561 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:41:34.673774 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39251
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34379
--webserver_port=38093
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:41:34.674549 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:34.674254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:39667)
0621 02:41:34.674381 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:34.674449 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:41:34.674846 15646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:41:34.993775 15649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:34.994047 15649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:34.994143 15649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:35.001060 15649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:35.001158 15649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:35.001266 15649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:41:35.010545 15649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38093
--tserver_master_addrs=127.0.97.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:35.011649 15649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:35.012856 15649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:35.025624 15654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:35.025681 15655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:35.025827 15657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:35.026537 15649 server_base.cc:1034] running on GCE node
I20260621 02:41:35.027130 15649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:35.028039 15649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:35.029260 15649 hybrid_clock.cc:648] HybridClock initialized: now 1782009695029210 us; error 61 us; skew 500 ppm
I20260621 02:41:35.029618 15649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:35 dist-test-slave-jkn7 krb5kdc[14201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:35.032655 15649 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:41:35.034139 15649 webserver.cc:460] Webserver started at http://127.0.97.3:38093/ using document root <none> and password file <none>
I20260621 02:41:35.034793 15649 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:35.034945 15649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:35.040056 15649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:41:35.042611 15664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:35.043848 15649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:41:35.044018 15649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "a1ef4019af0548d492258bfa132feb33"
format_stamp: "Formatted at 2026-06-21 02:41:29 on dist-test-slave-jkn7"
server_key: "86e3ba938e2060463b5909199e94a17b"
server_key_iv: "2493b1d2a48914760e3698d99bac4efb"
server_key_version: "encryptionkey@0"
I20260621 02:41:35.044961 15649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:35.069747 15649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:35.076807 15649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:35.077135 15649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:35.078732 15649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:35.080799 15649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:35.080950 15649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:35.081067 15649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:35.081152 15649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:35.119803 15649 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39251
I20260621 02:41:35.119861 15776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39251 every 8 connection(s)
I20260621 02:41:35.121877 15649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:41:35.126823 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:35.124998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:54901)
0621 02:41:35.125787 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:35.126007 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":377,"threads_started":1}
I20260621 02:41:35.127187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 15649
W20260621 02:41:35.129640 15777 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34379 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34379: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
I20260621 02:41:35.130450 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:35.130122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:42397)
0621 02:41:35.130268 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:35.130346 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:41:35.131281 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:35.130967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:36065)
0621 02:41:35.131101 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:35.131183 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:41:35.131497 15777 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 02:41:35.202913 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:35.202162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:34059)
0621 02:41:35.202607 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:35.202695 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":213,"threads_started":1}
I20260621 02:41:35.512949 15782 raft_consensus.cc:482] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:35.513228 15782 raft_consensus.cc:504] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:35.514926 15782 leader_election.cc:288] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737), a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251)
Jun 21 02:41:35 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Jun 21 02:41:35 dist-test-slave-jkn7 krb5kdc[14201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:41:35.537256 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:35.515150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41737 (local address 127.0.97.1:59259)
0621 02:41:35.515477 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:35.515521 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:35.516119 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:35.520704 (+ 4585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:35.520716 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:35.521593 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:35.522276 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:35.522295 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:35.523844 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:35.523849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:35.525142 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:35.525153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:35.526130 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:35.526140 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:35.526453 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:35.527300 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:35.527317 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:35.530322 (+ 3005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:35.534084 (+ 3762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:35.534089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:35.534092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:35.534566 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:35.535004 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:35.535008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:35.535011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:35.535183 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:35.535984 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:35.535987 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:35.536130 (+ 143us) client_negotiation.cc:769] Sending connection context
0621 02:41:35.536289 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 02:41:35.536795 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":157,"spinlock_wait_cycles":17152}
I20260621 02:41:35.537321 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:35.515769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60225 (local address 127.0.97.3:39251)
0621 02:41:35.518357 (+ 2588us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:35.518369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:35.518405 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:41:35.519880 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:35.519887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:35.520248 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:35.520517 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:35.522485 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:35.523663 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:35.525349 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:35.525937 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:35.526132 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:35.530217 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:35.530229 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:35.530250 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:35.530296 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:35.533139 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:35.534078 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:35.534086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:35.534094 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:35.534282 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:35.534750 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:35.534754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:35.534756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:35.535413 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:35.535560 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:35.536518 (+ 958us) server_negotiation.cc:299] Negotiation successful
0621 02:41:35.536903 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2418,"thread_start_us":2251,"threads_started":1}
I20260621 02:41:35.537376 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:35.515569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59259 (local address 127.0.97.2:41737)
0621 02:41:35.518615 (+ 3046us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:35.518625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:35.518654 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:41:35.520012 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:35.520020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:35.520357 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:35.520708 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:35.522488 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:35.523642 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:35.525281 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:35.525923 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:35.526103 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:35.530566 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:35.530572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:35.530583 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:35.530622 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:35.533929 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:35.534689 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:35.534699 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:35.534708 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:35.534866 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:35.535278 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:35.535281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:35.535283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:35.535844 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:35.536032 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:35.536548 (+ 516us) server_negotiation.cc:299] Negotiation successful
0621 02:41:35.536905 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2896,"thread_start_us":2786,"threads_started":1}
I20260621 02:41:35.537391 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:35.515232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39251 (local address 127.0.97.1:60225)
0621 02:41:35.517928 (+ 2696us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:35.517951 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:35.518091 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:35.520711 (+ 2620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:35.520716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:35.521594 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:35.522278 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:35.522297 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:35.523849 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:35.523853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:35.525149 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:35.525160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:35.526159 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:35.526165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:35.526450 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:35.527280 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:35.527302 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:35.529907 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:35.533310 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:35.533320 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:35.533334 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:35.533913 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:35.534404 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:35.534408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:35.534410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:35.534628 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:35.535555 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:35.535562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:35.535885 (+ 323us) client_negotiation.cc:769] Sending connection context
0621 02:41:35.536216 (+ 331us) client_negotiation.cc:240] Negotiation successful
0621 02:41:35.536945 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2530,"mutex_wait_us":444,"spinlock_wait_cycles":27136,"thread_start_us":2348,"threads_started":1}
W20260621 02:41:35.541404 15402 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
W20260621 02:41:35.541648 15400 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
I20260621 02:41:35.541813 15400 leader_election.cc:302] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83ae2e3426cb4de585d9c2297b823c8c; no voters: 0e0eddbc17954ae989cba507455893bb, a1ef4019af0548d492258bfa132feb33
I20260621 02:41:35.542250 15782 raft_consensus.cc:2726] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:41:35.676221 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:35.675348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:43837)
0621 02:41:35.675798 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:35.675921 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":221,"threads_started":1}
I20260621 02:41:36.132746 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:36.132010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:50719)
0621 02:41:36.132445 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:36.132543 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260621 02:41:36.204510 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:36.203810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:48869)
0621 02:41:36.204272 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:36.204354 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260621 02:41:36.678108 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:36.677160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:54621)
0621 02:41:36.677753 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:36.677893 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":369,"threads_started":1}
I20260621 02:41:37.134255 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:37.133558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:51467)
0621 02:41:37.134005 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:37.134084 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260621 02:41:37.206080 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:37.205415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:44457)
0621 02:41:37.205822 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:37.205895 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":202,"threads_started":1}
I20260621 02:41:37.314215 15792 raft_consensus.cc:482] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:37.314438 15792 raft_consensus.cc:504] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:37.315044 15792 leader_election.cc:288] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737), a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251)
W20260621 02:41:37.316047 15402 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
W20260621 02:41:37.316254 15400 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
I20260621 02:41:37.316373 15400 leader_election.cc:302] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83ae2e3426cb4de585d9c2297b823c8c; no voters: 0e0eddbc17954ae989cba507455893bb, a1ef4019af0548d492258bfa132feb33
I20260621 02:41:37.316588 15792 raft_consensus.cc:2726] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:41:37.679570 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:37.678891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:43623)
0621 02:41:37.679326 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:37.679404 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":216,"threads_started":1}
I20260621 02:41:38.135732 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:38.135059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:32783)
0621 02:41:38.135486 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:38.135569 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":236,"threads_started":1}
I20260621 02:41:38.207464 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:38.206830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:41419)
0621 02:41:38.207235 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:38.207310 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":215,"threads_started":1}
I20260621 02:41:38.681111 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:38.680439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:59633)
0621 02:41:38.680866 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:38.680940 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":206,"threads_started":1}
I20260621 02:41:39.137517 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:39.136672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:37769)
0621 02:41:39.137215 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:39.137350 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":276,"threads_started":1}
I20260621 02:41:39.168347 15798 raft_consensus.cc:482] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:39.168545 15798 raft_consensus.cc:504] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:39.169227 15798 leader_election.cc:288] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737), a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251)
W20260621 02:41:39.170280 15402 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
W20260621 02:41:39.170495 15400 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
I20260621 02:41:39.170636 15400 leader_election.cc:302] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83ae2e3426cb4de585d9c2297b823c8c; no voters: 0e0eddbc17954ae989cba507455893bb, a1ef4019af0548d492258bfa132feb33
I20260621 02:41:39.170928 15798 raft_consensus.cc:2726] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:41:39.209218 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:39.208397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:58447)
0621 02:41:39.208953 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:39.209029 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":238,"threads_started":1}
I20260621 02:41:39.682771 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:39.682035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:40733)
0621 02:41:39.682520 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:39.682606 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":275,"threads_started":1}
I20260621 02:41:40.138983 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:40.138348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:48029)
0621 02:41:40.138751 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:40.138837 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":225,"threads_started":1}
I20260621 02:41:40.210775 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:40.210108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:45573)
0621 02:41:40.210540 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:40.210622 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":236,"threads_started":1}
I20260621 02:41:40.684383 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:40.683670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:36199)
0621 02:41:40.684128 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:40.684205 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260621 02:41:41.140583 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:41.139885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:38655)
0621 02:41:41.140330 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:41.140409 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":230,"threads_started":1}
I20260621 02:41:41.212244 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:41.211592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:58697)
0621 02:41:41.211987 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:41.212061 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":214,"threads_started":1}
I20260621 02:41:41.686383 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:41.685393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:47869)
0621 02:41:41.685898 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:41.686148 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":231,"threads_started":1}
I20260621 02:41:42.142272 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:42.141519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:39889)
0621 02:41:42.141988 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:42.142065 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":220,"threads_started":1}
I20260621 02:41:42.213799 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:42.213105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:54523)
0621 02:41:42.213563 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:42.213648 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":256,"threads_started":1}
I20260621 02:41:42.688079 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:42.687392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:59777)
0621 02:41:42.687813 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:42.687893 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260621 02:41:43.143855 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:43.143163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:53311)
0621 02:41:43.143584 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:43.143660 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":234,"threads_started":1}
I20260621 02:41:43.215392 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:43.214651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:45795)
0621 02:41:43.215107 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:43.215196 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260621 02:41:43.689775 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:43.689003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:51031)
0621 02:41:43.689528 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:43.689611 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":222,"threads_started":1}
I20260621 02:41:44.044096 15813 raft_consensus.cc:482] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:44.044273 15813 raft_consensus.cc:504] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:44.044916 15813 leader_election.cc:288] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737), a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251)
W20260621 02:41:44.045876 15400 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
W20260621 02:41:44.046059 15402 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
I20260621 02:41:44.046222 15402 leader_election.cc:302] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83ae2e3426cb4de585d9c2297b823c8c; no voters: 0e0eddbc17954ae989cba507455893bb, a1ef4019af0548d492258bfa132feb33
I20260621 02:41:44.046491 15813 raft_consensus.cc:2726] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:41:44.145576 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:44.144766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:35343)
0621 02:41:44.145321 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:44.145416 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":253,"threads_started":1}
I20260621 02:41:44.217048 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:44.216310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:50047)
0621 02:41:44.216758 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:44.216858 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":218,"threads_started":1}
I20260621 02:41:44.691387 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:44.690606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:50313)
0621 02:41:44.691120 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:44.691212 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":234,"threads_started":1}
I20260621 02:41:45.147331 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:45.146531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:45255)
0621 02:41:45.147035 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:45.147138 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":258,"threads_started":1}
I20260621 02:41:45.218786 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:45.218080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:58837)
0621 02:41:45.218528 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:45.218611 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":239,"threads_started":1}
I20260621 02:41:45.693068 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:45.692364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:44465)
0621 02:41:45.692820 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:45.692909 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":240,"threads_started":1}
I20260621 02:41:46.149102 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:46.148280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:32809)
0621 02:41:46.148857 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:46.148944 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":332,"threads_started":1}
I20260621 02:41:46.220321 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:46.219636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:57229)
0621 02:41:46.220078 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:46.220153 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":252,"threads_started":1}
I20260621 02:41:46.694868 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:46.694013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:54299)
0621 02:41:46.694545 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:46.694639 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
I20260621 02:41:47.150671 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:47.149935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:56657)
0621 02:41:47.150417 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:47.150495 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":262,"threads_started":1}
I20260621 02:41:47.221946 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:47.221185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:55653)
0621 02:41:47.221698 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:47.221776 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":320,"threads_started":1}
I20260621 02:41:47.696440 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:47.695733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:48097)
0621 02:41:47.696186 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:47.696264 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":247,"threads_started":1}
I20260621 02:41:48.152329 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:48.151661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:34107)
0621 02:41:48.152095 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:48.152175 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
I20260621 02:41:48.223570 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:48.222848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:39481)
0621 02:41:48.223281 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:48.223381 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":241,"threads_started":1}
I20260621 02:41:48.698053 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:48.697276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:38907)
0621 02:41:48.697765 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:48.697852 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":217,"threads_started":1}
I20260621 02:41:49.153995 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:49.153210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:45221)
0621 02:41:49.153701 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:49.153787 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":283,"threads_started":1}
I20260621 02:41:49.225485 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:49.224731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.1:42381)
0621 02:41:49.225165 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:49.225249 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":212,"threads_started":1}
I20260621 02:41:49.699764 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:49.699107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.2:46619)
0621 02:41:49.699507 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:49.699590 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":207,"threads_started":1}
I20260621 02:41:49.928427 15832 raft_consensus.cc:482] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:49.928647 15832 raft_consensus.cc:504] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ae2e3426cb4de585d9c2297b823c8c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33399 } } peers { permanent_uuid: "0e0eddbc17954ae989cba507455893bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41737 } } peers { permanent_uuid: "a1ef4019af0548d492258bfa132feb33" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39251 } }
I20260621 02:41:49.929329 15832 leader_election.cc:288] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737), a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251)
W20260621 02:41:49.930279 15402 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1ef4019af0548d492258bfa132feb33 (127.0.97.3:39251): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
W20260621 02:41:49.930480 15400 leader_election.cc:341] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e0eddbc17954ae989cba507455893bb (127.0.97.2:41737): Not found: Tablet not found: 51b422fdbcb34f2f8fa07616c3f5d7ab
I20260621 02:41:49.930639 15400 leader_election.cc:302] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83ae2e3426cb4de585d9c2297b823c8c; no voters: 0e0eddbc17954ae989cba507455893bb, a1ef4019af0548d492258bfa132feb33
I20260621 02:41:49.930896 15832 raft_consensus.cc:2726] T 51b422fdbcb34f2f8fa07616c3f5d7ab P 83ae2e3426cb4de585d9c2297b823c8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:41:50.129868 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 15350
I20260621 02:41:50.149354 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:50.130995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.0.97.1:33399)
0621 02:41:50.131599 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:50.131606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:50.131632 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:41:50.131693 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:50.131699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:50.131959 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:50.132161 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:50.134246 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:50.135245 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:50.136494 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:50.137044 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:50.137179 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:50.139724 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:50.139741 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:50.139755 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:50.139795 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:50.146231 (+ 6436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:50.146987 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:50.146995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:50.146998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:50.147134 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:50.147546 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:50.147551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:50.147554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:50.147998 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:50.148294 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:50.148640 (+ 346us) server_negotiation.cc:299] Negotiation successful
0621 02:41:50.148915 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":281,"threads_started":1}
I20260621 02:41:50.150592 15510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:50.156087 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:50.155221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34379 (local address 127.0.97.3:35851)
0621 02:41:50.155737 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:50.155848 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":308,"threads_started":1}
I20260621 02:41:50.241947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 15350
I20260621 02:41:50.260016 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 15518
I20260621 02:41:50.273476 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:50.260797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.0.97.2:41737)
0621 02:41:50.261240 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:50.261245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:50.261262 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:41:50.261372 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:50.261376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:50.261436 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:50.261533 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:50.262764 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:50.263516 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:50.264710 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:50.265116 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:50.265207 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:50.267723 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:50.267728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:50.267732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:50.267765 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:50.270774 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:50.271472 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:50.271477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:50.271479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:50.271625 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:50.272020 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:50.272023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:50.272025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:50.272277 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:50.272498 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:50.272802 (+ 304us) server_negotiation.cc:299] Negotiation successful
0621 02:41:50.272976 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":243,"threads_started":1}
I20260621 02:41:50.274173 15641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:50.356648 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 15518
I20260621 02:41:50.374120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 15649
I20260621 02:41:50.387540 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:50.375016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.0.97.3:39251)
0621 02:41:50.375536 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:50.375541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:50.375558 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:41:50.375612 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:50.375615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:50.375684 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:50.375780 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:50.377100 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:50.378017 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:50.379201 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:50.379613 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:50.379713 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:50.382319 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:50.382324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:50.382328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:50.382380 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:50.385052 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:50.385775 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:50.385779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:50.385781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:50.385933 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:50.386247 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:50.386249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:50.386251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:50.386521 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:50.386722 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:50.386951 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 02:41:50.387120 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":257,"threads_started":1}
I20260621 02:41:50.388271 15772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:50.472203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 15649
2026-06-21T02:41:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24076 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:41:50 dist-test-slave-jkn7 krb5kdc[15843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:41:50 dist-test-slave-jkn7 krb5kdc[15843](info): set up 2 sockets
Jun 21 02:41:50 dist-test-slave-jkn7 krb5kdc[15843](info): commencing operation
krb5kdc: starting...
W20260621 02:41:51.565778 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:41:51 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:41:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:41:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:41:51.788791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46193
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:52.110066 15856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:52.110384 15856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:52.110540 15856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:52.117506 15856 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 02:41:52.117626 15856 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 02:41:52.117720 15856 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 02:41:52.117817 15856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:41:52.117915 15856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:52.118023 15856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:52.118109 15856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:41:52.118188 15856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:41:52.127480 15856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:52.128674 15856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:52.129961 15856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:41:52.141760 15856 server_base.cc:1034] running on GCE node
W20260621 02:41:52.141804 15861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:52.141755 15862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:52.141760 15864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:52.142923 15856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:52.144250 15856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:52.145432 15856 hybrid_clock.cc:648] HybridClock initialized: now 1782009712145366 us; error 30 us; skew 500 ppm
I20260621 02:41:52.145799 15856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:52 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:52.149098 15856 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:41:52.150604 15856 webserver.cc:460] Webserver started at http://127.0.97.62:34877/ using document root <none> and password file <none>
I20260621 02:41:52.151314 15856 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:52.151444 15856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:52.151764 15856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:52.154814 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "08b9a1b422254be6af9b80a526fb1e5f"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "3839e081361a1da2d6bc6cdb75c76970"
server_key_iv: "54738a225a829b312d013cbb7920ca10"
server_key_version: "encryptionkey@0"
I20260621 02:41:52.155623 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "08b9a1b422254be6af9b80a526fb1e5f"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "3839e081361a1da2d6bc6cdb75c76970"
server_key_iv: "54738a225a829b312d013cbb7920ca10"
server_key_version: "encryptionkey@0"
I20260621 02:41:52.160753 15856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:41:52.164417 15871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:52.165886 15856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:41:52.166087 15856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "08b9a1b422254be6af9b80a526fb1e5f"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "3839e081361a1da2d6bc6cdb75c76970"
server_key_iv: "54738a225a829b312d013cbb7920ca10"
server_key_version: "encryptionkey@0"
I20260621 02:41:52.166329 15856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:52.189828 15856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:52.194272 15856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:52.194558 15856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:52.212937 15856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46193
I20260621 02:41:52.212956 15922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46193 every 8 connection(s)
I20260621 02:41:52.214352 15856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:41:52.218014 15923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:52.220717 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 15856
I20260621 02:41:52.220944 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:41:52.221768 388 external_mini_cluster.cc:1428] Setting key 1213caab1c303788fc9646f15fed435a
I20260621 02:41:52.229851 15923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Bootstrap starting.
I20260621 02:41:52.232806 15923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:52.234141 15923 log.cc:824] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:41:52 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:52.237033 15923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: No bootstrap required, opened a new log
I20260621 02:41:52.244686 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:52.223876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.0.97.62:46193)
0621 02:41:52.224612 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:52.224624 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:52.224663 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:41:52.226323 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:52.226349 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:52.226733 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:52.227136 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:52.228584 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:52.229908 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:52.231327 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:52.231982 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:52.232171 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:52.236852 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:52.236860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:52.236873 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:52.236911 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:52.240072 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:52.240804 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:52.240810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:52.240816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:52.240989 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:52.241420 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:52.241426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:52.241430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:52.242147 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:52.242452 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:52.242837 (+ 385us) server_negotiation.cc:299] Negotiation successful
0621 02:41:52.243290 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":324,"threads_started":1}
I20260621 02:41:52.246371 15923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:41:52.246745 15923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:52.246861 15923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b9a1b422254be6af9b80a526fb1e5f, State: Initialized, Role: FOLLOWER
I20260621 02:41:52.247551 15923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:41:52.247829 15923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:41:52.247974 15923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:41:52.248162 15923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:52.250376 15923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:41:52.250856 15923 leader_election.cc:302] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08b9a1b422254be6af9b80a526fb1e5f; no voters:
I20260621 02:41:52.251413 15923 leader_election.cc:288] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:41:52.251562 15928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:41:52.252297 15928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 1 LEADER]: Becoming Leader. State: Replica: 08b9a1b422254be6af9b80a526fb1e5f, State: Running, Role: LEADER
I20260621 02:41:52.252902 15928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:41:52.253536 15923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:41:52.255990 15930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08b9a1b422254be6af9b80a526fb1e5f. Latest consensus state: current_term: 1 leader_uuid: "08b9a1b422254be6af9b80a526fb1e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } } }
I20260621 02:41:52.256361 15930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:52.256124 15929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08b9a1b422254be6af9b80a526fb1e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } } }
I20260621 02:41:52.256537 15929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: This master's current role is: LEADER
I20260621 02:41:52.258798 15934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:41:52.263644 15934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:41:52.272097 15934 catalog_manager.cc:1347] Generated new cluster ID: 6e2b127ca44a4e8fb74a1f36c10df654
I20260621 02:41:52.272236 15934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:41:52.284816 15934 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:41:52.286327 15934 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:41:52.292888 15934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Generated new TSK 0
I20260621 02:41:52.293710 15934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:41:52.351084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46193
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:52.680590 15949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:52.680896 15949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:52.681058 15949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:52.688524 15949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:52.688645 15949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:52.688827 15949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:41:52.698976 15949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:52.700328 15949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:52.701759 15949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:52.715569 15954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:52.715543 15955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:52.715652 15957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:52.716390 15949 server_base.cc:1034] running on GCE node
I20260621 02:41:52.717052 15949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:52.718238 15949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:52.719524 15949 hybrid_clock.cc:648] HybridClock initialized: now 1782009712719451 us; error 72 us; skew 500 ppm
I20260621 02:41:52.719901 15949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:52 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:52.723275 15949 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:41:52.725240 15949 webserver.cc:460] Webserver started at http://127.0.97.1:43487/ using document root <none> and password file <none>
I20260621 02:41:52.725968 15949 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:52.726073 15949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:52.726325 15949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:52.729213 15949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "83fb564fab5a4aae87b0aa8a062c1982"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "22380482e646eedcb21f8d741036c084"
server_key_iv: "4974de58057a5708d31b7cfe15943c41"
server_key_version: "encryptionkey@0"
I20260621 02:41:52.730003 15949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "83fb564fab5a4aae87b0aa8a062c1982"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "22380482e646eedcb21f8d741036c084"
server_key_iv: "4974de58057a5708d31b7cfe15943c41"
server_key_version: "encryptionkey@0"
I20260621 02:41:52.735168 15949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:41:52.739048 15964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:52.740450 15949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:41:52.740597 15949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "83fb564fab5a4aae87b0aa8a062c1982"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "22380482e646eedcb21f8d741036c084"
server_key_iv: "4974de58057a5708d31b7cfe15943c41"
server_key_version: "encryptionkey@0"
I20260621 02:41:52.740772 15949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:52.754782 15949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:52.759449 15949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:52.759699 15949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:52.760632 15949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:52.761973 15949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:52.762048 15949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:52.762136 15949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:52.762179 15949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:52.795550 15949 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44271
I20260621 02:41:52.795571 16076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44271 every 8 connection(s)
I20260621 02:41:52.796830 15949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:41:52.805882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 15949
I20260621 02:41:52.806109 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:41:52.806737 388 external_mini_cluster.cc:1428] Setting key 08122ea8cc6cc4f69835a75e3a1ceaae
Jun 21 02:41:52 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:52.816648 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:52.799329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54763 (local address 127.0.97.62:46193)
0621 02:41:52.799767 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:52.799772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:52.800591 (+ 819us) server_negotiation.cc:407] Connection header received
0621 02:41:52.802430 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:52.802436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:52.802540 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:52.802660 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:52.804089 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:52.804810 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:52.806466 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:52.806917 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:52.807055 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:52.811086 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:52.811091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:52.811094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:52.811125 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:52.813720 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:52.814425 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:52.814437 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:52.814439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:52.814558 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:52.814995 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:52.815001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:52.815005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:52.815252 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:52.815405 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:52.816073 (+ 668us) server_negotiation.cc:299] Negotiation successful
0621 02:41:52.816281 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":247,"threads_started":1}
I20260621 02:41:52.817655 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:52.799779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:54763)
0621 02:41:52.800443 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:52.800482 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:52.802177 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:52.802885 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:52.802894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:52.803391 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:52.803915 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:52.803932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:52.804960 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:52.804975 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:52.806329 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:52.806340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:52.807140 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:52.807149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:52.807491 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:52.808202 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:52.808224 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:52.810880 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:52.813864 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:52.813872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:52.813887 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:52.814298 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:52.814676 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:52.814681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:52.814684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:52.814866 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:52.815379 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:52.815387 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:52.815702 (+ 315us) client_negotiation.cc:769] Sending connection context
0621 02:41:52.815956 (+ 254us) client_negotiation.cc:240] Negotiation successful
0621 02:41:52.816456 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":269,"threads_started":1}
I20260621 02:41:52.819275 16077 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46193
I20260621 02:41:52.819607 16077 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:52.820283 16077 heartbeater.cc:511] Master 127.0.97.62:46193 requested a full tablet report, sending...
I20260621 02:41:52.822552 15888 ts_manager.cc:194] Registered new tserver with Master: 83fb564fab5a4aae87b0aa8a062c1982 (127.0.97.1:44271)
I20260621 02:41:52.824731 15888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54763
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:41:52.860240 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46193
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:53.177449 16083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:53.177727 16083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:53.177821 16083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:53.184868 16083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:53.184991 16083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:53.185105 16083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:41:53.194770 16083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:53.196027 16083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:53.197261 16083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:53.210179 16088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:53.210142 16089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:53.210212 16091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:53.211174 16083 server_base.cc:1034] running on GCE node
I20260621 02:41:53.211826 16083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:53.213030 16083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:53.214272 16083 hybrid_clock.cc:648] HybridClock initialized: now 1782009713214172 us; error 111 us; skew 500 ppm
I20260621 02:41:53.214630 16083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:53 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:53.218225 16083 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:41:53.219693 16083 webserver.cc:460] Webserver started at http://127.0.97.2:40133/ using document root <none> and password file <none>
I20260621 02:41:53.220367 16083 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:53.220497 16083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:53.220816 16083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:53.223706 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "530274f8484b4b4aaa55bc4015b2f5ba"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "7ee98eee0f3e62a0f6ecfe6d9755ed61"
server_key_iv: "098c64cd9ba51adc0fb0e72ca9c9d64c"
server_key_version: "encryptionkey@0"
I20260621 02:41:53.224499 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "530274f8484b4b4aaa55bc4015b2f5ba"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "7ee98eee0f3e62a0f6ecfe6d9755ed61"
server_key_iv: "098c64cd9ba51adc0fb0e72ca9c9d64c"
server_key_version: "encryptionkey@0"
I20260621 02:41:53.229769 16083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:41:53.233367 16098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:53.234725 16083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:41:53.234913 16083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "530274f8484b4b4aaa55bc4015b2f5ba"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "7ee98eee0f3e62a0f6ecfe6d9755ed61"
server_key_iv: "098c64cd9ba51adc0fb0e72ca9c9d64c"
server_key_version: "encryptionkey@0"
I20260621 02:41:53.235131 16083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:53.257117 16083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:53.261152 16083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:53.261453 16083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:53.262405 16083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:53.263720 16083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:53.263797 16083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:53.263965 16083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:53.264038 16083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:53.297108 16083 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40965
I20260621 02:41:53.297154 16210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40965 every 8 connection(s)
I20260621 02:41:53.298509 16083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:41:53.302320 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 16083
I20260621 02:41:53.302549 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:41:53.303089 388 external_mini_cluster.cc:1428] Setting key 54c3a4c42514488adcc6d447bd7fc74b
Jun 21 02:41:53 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:53.320451 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.301326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59401 (local address 127.0.97.62:46193)
0621 02:41:53.301521 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:53.301526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:53.302829 (+ 1303us) server_negotiation.cc:407] Connection header received
0621 02:41:53.304843 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:53.304848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:53.304933 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:53.305071 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:53.306614 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.307310 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.308825 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.309254 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.309439 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.313052 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:53.313061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:53.313069 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:53.313116 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:53.316777 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:53.317564 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:53.317574 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:53.317580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:53.317741 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:53.318176 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:53.318184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:53.318190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:53.318565 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:53.318712 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:53.319560 (+ 848us) server_negotiation.cc:299] Negotiation successful
0621 02:41:53.319924 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:41:53.321161 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.301774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:59401)
0621 02:41:53.302636 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:53.302688 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:53.304574 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:53.305363 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:53.305371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:53.305888 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:53.306423 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.306444 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.307467 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.307482 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.308673 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.308686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.309531 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.309539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.309759 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.310441 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:53.310464 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:53.312836 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:53.316919 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:53.316926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:53.316937 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:53.317444 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:53.317850 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:53.317853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:53.317855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:53.318044 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:53.318686 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:53.318692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:53.319042 (+ 350us) client_negotiation.cc:769] Sending connection context
0621 02:41:53.319358 (+ 316us) client_negotiation.cc:240] Negotiation successful
0621 02:41:53.319980 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":380,"threads_started":1}
I20260621 02:41:53.322681 16211 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46193
I20260621 02:41:53.322985 16211 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:53.323776 16211 heartbeater.cc:511] Master 127.0.97.62:46193 requested a full tablet report, sending...
I20260621 02:41:53.325206 15888 ts_manager.cc:194] Registered new tserver with Master: 530274f8484b4b4aaa55bc4015b2f5ba (127.0.97.2:40965)
I20260621 02:41:53.326112 15888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59401
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:41:53.356206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46193
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:41:53.672441 16216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:41:53.672694 16216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:41:53.672785 16216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:41:53.679894 16216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:41:53.679989 16216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:41:53.680095 16216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:41:53.689946 16216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:41:53.691087 16216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:41:53.692319 16216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:41:53.704882 16224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:53.704926 16222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:41:53.704953 16221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:41:53.706259 16216 server_base.cc:1034] running on GCE node
I20260621 02:41:53.706914 16216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:41:53.707907 16216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:41:53.709102 16216 hybrid_clock.cc:648] HybridClock initialized: now 1782009713709070 us; error 33 us; skew 500 ppm
I20260621 02:41:53.709476 16216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:41:53 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:41:53.712917 16216 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:41:53.714255 16216 webserver.cc:460] Webserver started at http://127.0.97.3:44547/ using document root <none> and password file <none>
I20260621 02:41:53.714876 16216 fs_manager.cc:359] Metadata directory not provided
I20260621 02:41:53.714977 16216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:41:53.715226 16216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:41:53.718048 16216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "9553268e49b945f1a2f5e2b1af147775"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "af5ed90de37367e1a42a7eac9c3e4f99"
server_key_iv: "1e541388d21c46ede62c0ae9d5709005"
server_key_version: "encryptionkey@0"
I20260621 02:41:53.718735 16216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9553268e49b945f1a2f5e2b1af147775"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "af5ed90de37367e1a42a7eac9c3e4f99"
server_key_iv: "1e541388d21c46ede62c0ae9d5709005"
server_key_version: "encryptionkey@0"
I20260621 02:41:53.724042 16216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:41:53.727614 16231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:53.728907 16216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:41:53.729085 16216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "9553268e49b945f1a2f5e2b1af147775"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "af5ed90de37367e1a42a7eac9c3e4f99"
server_key_iv: "1e541388d21c46ede62c0ae9d5709005"
server_key_version: "encryptionkey@0"
I20260621 02:41:53.729283 16216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:41:53.745556 16216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:41:53.749174 16216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:41:53.749471 16216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:41:53.750365 16216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:41:53.751605 16216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:41:53.751694 16216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:53.751829 16216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:41:53.751879 16216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:41:53.785378 16216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46117
I20260621 02:41:53.785405 16343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46117 every 8 connection(s)
I20260621 02:41:53.786618 16216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:41:53.789239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 16216
I20260621 02:41:53.789542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:41:53.790010 388 external_mini_cluster.cc:1428] Setting key 8574f327c9594dcb8e005486b61465b3
Jun 21 02:41:53 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:41:53.807160 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.789151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60593 (local address 127.0.97.62:46193)
0621 02:41:53.789403 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:53.789410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:53.790651 (+ 1241us) server_negotiation.cc:407] Connection header received
0621 02:41:53.792648 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:53.792655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:53.792737 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:53.792861 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:53.794312 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.795084 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.797341 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.797736 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.797830 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.801702 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:53.801708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:53.801711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:53.801742 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:53.804433 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:53.805139 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:53.805146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:53.805150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:53.805331 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:53.805763 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:53.805766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:53.805769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:53.806048 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:53.806157 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:53.806764 (+ 607us) server_negotiation.cc:299] Negotiation successful
0621 02:41:53.806925 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260621 02:41:53.808028 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.789587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:60593)
0621 02:41:53.790468 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:53.790528 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:53.792383 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:53.793067 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:53.793075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:53.793635 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:41:53.794168 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.794184 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.795224 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.795229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.797162 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.797177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.797965 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.797973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.798245 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.798968 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:41:53.798996 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:41:53.801491 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:41:53.804567 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:53.804576 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:53.804590 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:53.804976 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:53.805453 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:41:53.805457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:41:53.805459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:41:53.805648 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:41:53.806150 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:41:53.806155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:41:53.806433 (+ 278us) client_negotiation.cc:769] Sending connection context
0621 02:41:53.806658 (+ 225us) client_negotiation.cc:240] Negotiation successful
0621 02:41:53.806991 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":375,"threads_started":1}
I20260621 02:41:53.809438 16344 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46193
I20260621 02:41:53.809743 16344 heartbeater.cc:464] Registering TS with master...
I20260621 02:41:53.810411 16344 heartbeater.cc:511] Master 127.0.97.62:46193 requested a full tablet report, sending...
I20260621 02:41:53.811748 15888 ts_manager.cc:194] Registered new tserver with Master: 9553268e49b945f1a2f5e2b1af147775 (127.0.97.3:46117)
I20260621 02:41:53.812469 15888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60593
I20260621 02:41:53.817267 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:41:53.827756 16077 heartbeater.cc:503] Master 127.0.97.62:46193 was elected leader, sending a full tablet report...
I20260621 02:41:53.833913 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.821695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.97.62:46193)
0621 02:41:53.821875 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:53.821882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:53.822284 (+ 402us) server_negotiation.cc:407] Connection header received
0621 02:41:53.822470 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:53.822475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:53.822551 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:53.822649 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:53.823999 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.824687 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.826062 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.826483 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.826745 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.829039 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:53.829045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:53.829050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:53.829089 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:53.831442 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:53.832130 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:53.832135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:53.832139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:53.832257 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:53.832661 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:53.832665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:53.832667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:53.832963 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:53.833170 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:53.833540 (+ 370us) server_negotiation.cc:299] Negotiation successful
0621 02:41:53.833702 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:41:53.837919 15888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:41:53.844856 15888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:41:53.867967 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.855801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46117 (local address 127.0.0.1:36510)
0621 02:41:53.856552 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:53.856601 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:53.856783 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:53.858917 (+ 2134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:53.858926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:53.858963 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:53.859441 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.859457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.862327 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.862336 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.865276 (+ 2940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.865310 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.866997 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.867004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.867200 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.867244 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 02:41:53.867364 (+ 120us) client_negotiation.cc:240] Negotiation successful
0621 02:41:53.867563 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":363,"threads_started":1}
I20260621 02:41:53.868315 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.856044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.0.97.3:46117)
0621 02:41:53.857516 (+ 1472us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:53.857526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:53.857562 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:41:53.857665 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:53.857675 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:53.857939 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:53.859734 (+ 1795us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:53.860095 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.862136 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.865491 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.866461 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.866573 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.866804 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:53.867785 (+ 981us) server_negotiation.cc:299] Negotiation successful
0621 02:41:53.867953 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":1055,"threads_started":1}
I20260621 02:41:53.872440 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.856400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.0.97.2:40965)
0621 02:41:53.865253 (+ 8853us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:53.865266 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:53.865315 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:41:53.865424 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:53.865431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:53.865644 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:53.865834 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:53.866497 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.868015 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.870093 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.871403 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.871560 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.871749 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:53.871971 (+ 222us) server_negotiation.cc:299] Negotiation successful
0621 02:41:53.872122 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8639,"thread_start_us":8511,"threads_started":1}
I20260621 02:41:53.872496 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.855814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40965 (local address 127.0.0.1:44890)
0621 02:41:53.858504 (+ 2690us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:53.858526 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:53.858683 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:53.865821 (+ 7138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:53.865826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:53.865851 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:53.866283 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.866300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.868321 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.868326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.869941 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.869952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.871619 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.871626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.871792 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.871825 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:41:53.871932 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 02:41:53.872059 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2533,"mutex_wait_us":3,"thread_start_us":2394,"threads_started":1}
I20260621 02:41:53.872156 16279 tablet_service.cc:1467] Processing CreateTablet for tablet f78601d6673e4007a836ae63aa330204 (DEFAULT_TABLE table=test-table [id=a1f79bbaf29a4c6db61bdd54959b0609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:53.873777 16279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f78601d6673e4007a836ae63aa330204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:53.875975 16146 tablet_service.cc:1467] Processing CreateTablet for tablet f78601d6673e4007a836ae63aa330204 (DEFAULT_TABLE table=test-table [id=a1f79bbaf29a4c6db61bdd54959b0609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:53.877276 16146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f78601d6673e4007a836ae63aa330204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:53.878455 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.857037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:41636)
0621 02:41:53.866491 (+ 9454us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:53.866511 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:53.866638 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:53.867453 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:53.867458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:53.867482 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:53.867872 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.867888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.870645 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.870650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.872660 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.872675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.877540 (+ 4865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.877547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.877870 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.877905 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:41:53.878000 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:41:53.878125 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9303,"thread_start_us":9674,"threads_started":1}
I20260621 02:41:53.879201 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.857655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.0.97.1:44271)
0621 02:41:53.865671 (+ 8016us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:53.865680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:53.866601 (+ 921us) server_negotiation.cc:407] Connection header received
0621 02:41:53.866793 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:53.866800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:53.867024 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:53.867201 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:53.868072 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.870477 (+ 2405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.872848 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.877273 (+ 4425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.877457 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.877887 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:53.878717 (+ 830us) server_negotiation.cc:299] Negotiation successful
0621 02:41:53.878898 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7846,"thread_start_us":7658,"threads_started":1}
I20260621 02:41:53.881932 16012 tablet_service.cc:1467] Processing CreateTablet for tablet f78601d6673e4007a836ae63aa330204 (DEFAULT_TABLE table=test-table [id=a1f79bbaf29a4c6db61bdd54959b0609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:41:53.883147 16012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f78601d6673e4007a836ae63aa330204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:41:53.889441 16362 tablet_bootstrap.cc:492] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: Bootstrap starting.
I20260621 02:41:53.892256 16363 tablet_bootstrap.cc:492] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: Bootstrap starting.
I20260621 02:41:53.892989 16362 tablet_bootstrap.cc:654] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:53.893771 16364 tablet_bootstrap.cc:492] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: Bootstrap starting.
I20260621 02:41:53.894336 16362 log.cc:824] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:53.895553 16363 tablet_bootstrap.cc:654] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:53.896911 16363 log.cc:824] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:53.897018 16364 tablet_bootstrap.cc:654] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: Neither blocks nor log segments found. Creating new log.
I20260621 02:41:53.897550 16362 tablet_bootstrap.cc:492] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: No bootstrap required, opened a new log
I20260621 02:41:53.897861 16362 ts_tablet_manager.cc:1397] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260621 02:41:53.898255 16364 log.cc:824] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: Log is configured to *not* fsync() on all Append() calls
I20260621 02:41:53.899646 16363 tablet_bootstrap.cc:492] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: No bootstrap required, opened a new log
I20260621 02:41:53.899894 16363 ts_tablet_manager.cc:1397] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260621 02:41:53.907308 16363 raft_consensus.cc:348] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } }
I20260621 02:41:53.907670 16363 raft_consensus.cc:374] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:53.907774 16363 raft_consensus.cc:729] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530274f8484b4b4aaa55bc4015b2f5ba, State: Initialized, Role: FOLLOWER
I20260621 02:41:53.908144 16364 tablet_bootstrap.cc:492] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: No bootstrap required, opened a new log
I20260621 02:41:53.908418 16364 ts_tablet_manager.cc:1397] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260621 02:41:53.908417 16363 consensus_queue.cc:260] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } }
I20260621 02:41:53.909518 16211 heartbeater.cc:503] Master 127.0.97.62:46193 was elected leader, sending a full tablet report...
I20260621 02:41:53.910048 16362 raft_consensus.cc:348] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } }
I20260621 02:41:53.910485 16362 raft_consensus.cc:374] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:53.910591 16362 raft_consensus.cc:729] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553268e49b945f1a2f5e2b1af147775, State: Initialized, Role: FOLLOWER
I20260621 02:41:53.911327 16362 consensus_queue.cc:260] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } }
I20260621 02:41:53.911887 16363 ts_tablet_manager.cc:1428] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260621 02:41:53.912830 16344 heartbeater.cc:503] Master 127.0.97.62:46193 was elected leader, sending a full tablet report...
I20260621 02:41:53.913430 16362 ts_tablet_manager.cc:1428] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260621 02:41:53.918133 16364 raft_consensus.cc:348] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } }
I20260621 02:41:53.918455 16364 raft_consensus.cc:374] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:41:53.918568 16364 raft_consensus.cc:729] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fb564fab5a4aae87b0aa8a062c1982, State: Initialized, Role: FOLLOWER
I20260621 02:41:53.919086 16364 consensus_queue.cc:260] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } }
I20260621 02:41:53.920581 16364 ts_tablet_manager.cc:1428] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260621 02:41:53.981935 16368 raft_consensus.cc:482] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:41:53.982196 16368 raft_consensus.cc:504] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } }
I20260621 02:41:53.983734 16368 leader_election.cc:288] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553268e49b945f1a2f5e2b1af147775 (127.0.97.3:46117), 83fb564fab5a4aae87b0aa8a062c1982 (127.0.97.1:44271)
I20260621 02:41:53.990854 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.984158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40413 (local address 127.0.97.3:46117)
0621 02:41:53.984337 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:53.984341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:53.984996 (+ 655us) server_negotiation.cc:407] Connection header received
0621 02:41:53.985265 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:53.985272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:53.985380 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:53.985458 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:53.986152 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.986877 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.989113 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.989931 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.990079 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.990188 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:53.990474 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 02:41:53.990594 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:41:53.990954 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.985263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53595 (local address 127.0.97.1:44271)
0621 02:41:53.985475 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:53.985482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:53.985658 (+ 176us) server_negotiation.cc:407] Connection header received
0621 02:41:53.985861 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:53.985871 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:53.985931 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:53.986022 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:41:53.986515 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.987200 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.989274 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.990037 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.990180 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.990279 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:53.990576 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 02:41:53.990698 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 02:41:53.991058 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.984157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46117 (local address 127.0.97.2:40413)
0621 02:41:53.984869 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:53.984887 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:53.985040 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:53.985626 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:53.985629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:53.985650 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:53.986000 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.986008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.987017 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.987024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.988963 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.988974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.990137 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.990144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.990303 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.990325 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:41:53.990448 (+ 123us) client_negotiation.cc:240] Negotiation successful
0621 02:41:53.990572 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":25,"spinlock_wait_cycles":16128,"thread_start_us":388,"threads_started":1}
I20260621 02:41:53.991113 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:53.984949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.97.2:53595)
0621 02:41:53.985558 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0621 02:41:53.985571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:41:53.985725 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:41:53.986034 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:41:53.986038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:41:53.986059 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:41:53.986363 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.986370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.987349 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.987354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.989131 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:41:53.989142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:41:53.990258 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:41:53.990266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:41:53.990429 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:53.990451 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:41:53.990550 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 02:41:53.990668 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":35,"thread_start_us":267,"threads_started":1}
I20260621 02:41:53.992215 16299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f78601d6673e4007a836ae63aa330204" candidate_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553268e49b945f1a2f5e2b1af147775" is_pre_election: true
I20260621 02:41:53.992215 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f78601d6673e4007a836ae63aa330204" candidate_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fb564fab5a4aae87b0aa8a062c1982" is_pre_election: true
I20260621 02:41:53.992791 16032 raft_consensus.cc:2454] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530274f8484b4b4aaa55bc4015b2f5ba in term 0.
I20260621 02:41:53.992791 16299 raft_consensus.cc:2454] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530274f8484b4b4aaa55bc4015b2f5ba in term 0.
I20260621 02:41:53.993485 16102 leader_election.cc:302] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530274f8484b4b4aaa55bc4015b2f5ba, 83fb564fab5a4aae87b0aa8a062c1982; no voters:
I20260621 02:41:53.993818 16368 raft_consensus.cc:2781] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:41:53.993964 16368 raft_consensus.cc:482] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:41:53.994073 16368 raft_consensus.cc:3037] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:53.996169 16368 raft_consensus.cc:504] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } }
I20260621 02:41:53.996752 16368 leader_election.cc:288] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [CANDIDATE]: Term 1 election: Requested vote from peers 9553268e49b945f1a2f5e2b1af147775 (127.0.97.3:46117), 83fb564fab5a4aae87b0aa8a062c1982 (127.0.97.1:44271)
I20260621 02:41:53.997177 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f78601d6673e4007a836ae63aa330204" candidate_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fb564fab5a4aae87b0aa8a062c1982"
I20260621 02:41:53.997159 16299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f78601d6673e4007a836ae63aa330204" candidate_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553268e49b945f1a2f5e2b1af147775"
I20260621 02:41:53.997388 16299 raft_consensus.cc:3037] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:53.997388 16032 raft_consensus.cc:3037] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:41:54.000512 16299 raft_consensus.cc:2454] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530274f8484b4b4aaa55bc4015b2f5ba in term 1.
I20260621 02:41:54.000515 16032 raft_consensus.cc:2454] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530274f8484b4b4aaa55bc4015b2f5ba in term 1.
I20260621 02:41:54.000890 16102 leader_election.cc:302] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530274f8484b4b4aaa55bc4015b2f5ba, 9553268e49b945f1a2f5e2b1af147775; no voters:
I20260621 02:41:54.001137 16368 raft_consensus.cc:2781] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:41:54.001585 16368 raft_consensus.cc:686] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 1 LEADER]: Becoming Leader. State: Replica: 530274f8484b4b4aaa55bc4015b2f5ba, State: Running, Role: LEADER
I20260621 02:41:54.002029 16368 consensus_queue.cc:237] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } }
I20260621 02:41:54.006667 15888 catalog_manager.cc:5495] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 530274f8484b4b4aaa55bc4015b2f5ba (127.0.97.2). New cstate: current_term: 1 leader_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } health_report { overall_health: UNKNOWN } } }
I20260621 02:41:54.030153 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:54.025035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.0.97.2:40965)
0621 02:41:54.025229 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:54.025234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:54.025253 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:41:54.025522 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:54.025526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:54.025623 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:54.025745 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:41:54.026090 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:54.026806 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:54.028256 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:54.028630 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:54.028717 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:54.029160 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:41:54.029677 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:41:54.029783 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:54.029879 (+ 96us) server_negotiation.cc:299] Negotiation successful
0621 02:41:54.029940 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 02:41:54.038157 16345 tablet.cc:2369] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:41:54.038211 16032 raft_consensus.cc:1261] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [term 1 FOLLOWER]: Refusing update from remote peer 530274f8484b4b4aaa55bc4015b2f5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:54.038209 16299 raft_consensus.cc:1261] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [term 1 FOLLOWER]: Refusing update from remote peer 530274f8484b4b4aaa55bc4015b2f5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:41:54.039186 16368 consensus_queue.cc:1035] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:41:54.039497 16373 consensus_queue.cc:1035] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:41:54.049222 16078 tablet.cc:2369] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:41:54.050659 16376 mvcc.cc:205] Tried to move back new op lower bound from 7299111788690001920 to 7299111788555976704. Current Snapshot: MvccSnapshot[applied={T|T < 7299111788690001920}]
I20260621 02:41:54.065514 15887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:41:54.065838 15887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:41:54.069622 15887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Sending DeleteTablet for 3 replicas of tablet f78601d6673e4007a836ae63aa330204
I20260621 02:41:54.070806 16012 tablet_service.cc:1514] Processing DeleteTablet for tablet f78601d6673e4007a836ae63aa330204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41636
I20260621 02:41:54.070775 16146 tablet_service.cc:1514] Processing DeleteTablet for tablet f78601d6673e4007a836ae63aa330204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44890
I20260621 02:41:54.071154 16279 tablet_service.cc:1514] Processing DeleteTablet for tablet f78601d6673e4007a836ae63aa330204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:41:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36510
I20260621 02:41:54.071792 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 15949
I20260621 02:41:54.083079 16388 tablet_replica.cc:330] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: stopping tablet replica
I20260621 02:41:54.083583 16388 raft_consensus.cc:2229] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:41:54.084252 16388 raft_consensus.cc:2258] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:54.088927 16386 tablet_replica.cc:330] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: stopping tablet replica
I20260621 02:41:54.089143 16387 tablet_replica.cc:330] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: stopping tablet replica
I20260621 02:41:54.089421 16386 raft_consensus.cc:2229] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:41:54.089576 16387 raft_consensus.cc:2229] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:41:54.089941 16386 raft_consensus.cc:2258] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:54.090266 16387 raft_consensus.cc:2258] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:41:54.090736 16388 ts_tablet_manager.cc:1905] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:54.092188 16387 ts_tablet_manager.cc:1905] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:54.093137 16386 ts_tablet_manager.cc:1905] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:41:54.097687 16388 ts_tablet_manager.cc:1918] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:54.097906 16388 log.cc:1196] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/f78601d6673e4007a836ae63aa330204
I20260621 02:41:54.098472 16388 ts_tablet_manager.cc:1939] T f78601d6673e4007a836ae63aa330204 P 9553268e49b945f1a2f5e2b1af147775: Deleting consensus metadata
I20260621 02:41:54.099442 16387 ts_tablet_manager.cc:1918] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:54.099658 16387 log.cc:1196] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/f78601d6673e4007a836ae63aa330204
I20260621 02:41:54.100236 16387 ts_tablet_manager.cc:1939] T f78601d6673e4007a836ae63aa330204 P 530274f8484b4b4aaa55bc4015b2f5ba: Deleting consensus metadata
I20260621 02:41:54.100415 15875 catalog_manager.cc:4841] TS 9553268e49b945f1a2f5e2b1af147775 (127.0.97.3:46117): tablet f78601d6673e4007a836ae63aa330204 (table test-table [id=a1f79bbaf29a4c6db61bdd54959b0609]) successfully deleted
I20260621 02:41:54.101909 15874 catalog_manager.cc:4841] TS 530274f8484b4b4aaa55bc4015b2f5ba (127.0.97.2:40965): tablet f78601d6673e4007a836ae63aa330204 (table test-table [id=a1f79bbaf29a4c6db61bdd54959b0609]) successfully deleted
I20260621 02:41:54.102052 16386 ts_tablet_manager.cc:1918] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:41:54.102209 16386 log.cc:1196] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/f78601d6673e4007a836ae63aa330204
I20260621 02:41:54.102672 16386 ts_tablet_manager.cc:1939] T f78601d6673e4007a836ae63aa330204 P 83fb564fab5a4aae87b0aa8a062c1982: Deleting consensus metadata
I20260621 02:41:54.103982 15875 catalog_manager.cc:4841] TS 83fb564fab5a4aae87b0aa8a062c1982 (127.0.97.1:44271): tablet f78601d6673e4007a836ae63aa330204 (table test-table [id=a1f79bbaf29a4c6db61bdd54959b0609]) successfully deleted
Jun 21 02:41:54 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:41:54.113910 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:54.073085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.97.1:44271)
0621 02:41:54.073325 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:54.073335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:54.085241 (+ 11906us) server_negotiation.cc:407] Connection header received
0621 02:41:54.085548 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:54.085557 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:54.085691 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:54.085785 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:54.090369 (+ 4584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:54.091497 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:54.100906 (+ 9409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:54.101635 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:54.101739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:54.107100 (+ 5361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:54.107115 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:54.107134 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:54.107183 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:54.110785 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:54.111563 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:54.111573 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:54.111603 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:54.111763 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:54.112267 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:54.112273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:54.112276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:54.112715 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:54.113012 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:54.113318 (+ 306us) server_negotiation.cc:299] Negotiation successful
0621 02:41:54.113539 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:41:54.114588 16072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:54.209203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 15949
I20260621 02:41:54.232946 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 16083
Jun 21 02:41:54 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:41:54.248862 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:54.233791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.0.97.2:40965)
0621 02:41:54.233985 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:54.233990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:54.234009 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:41:54.234176 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:54.234180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:54.234241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:54.234368 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:54.235670 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:54.236439 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:54.237949 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:54.238373 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:54.238534 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:54.242883 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:54.242896 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:54.242911 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:54.242954 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:54.246070 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:54.246762 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:54.246777 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:54.246780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:54.246912 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:54.247290 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:54.247296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:54.247299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:54.247660 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:54.247993 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:54.248293 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 02:41:54.248508 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:41:54.249684 16206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:54.360731 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 16083
I20260621 02:41:54.378815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 16216
Jun 21 02:41:54 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:41:54.393882 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:41:54.379658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.0.97.3:46117)
0621 02:41:54.379937 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0621 02:41:54.379942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:41:54.379960 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:41:54.380110 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:41:54.380115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:41:54.380178 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:41:54.380259 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:41:54.381561 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:54.382270 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:54.383552 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:41:54.383919 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:41:54.384009 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:41:54.388377 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:41:54.388391 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:41:54.388432 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:41:54.388472 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:41:54.391285 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:54.391938 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:54.391946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:54.391948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:54.392097 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:41:54.392450 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:41:54.392453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:41:54.392455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:41:54.392780 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:41:54.392990 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:41:54.393478 (+ 488us) server_negotiation.cc:299] Negotiation successful
0621 02:41:54.393647 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:41:54.394523 16339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:54.484958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 16216
I20260621 02:41:54.503333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 15856
I20260621 02:41:54.504429 15918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:41:54.613778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 15856
I20260621 02:42:14.635334 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46193
--webserver_interface=127.0.97.62
--webserver_port=34877
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:42:14.959563 16394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:14.960028 16394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:14.960220 16394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:14.967692 16394 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 02:42:14.967803 16394 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 02:42:14.967877 16394 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 02:42:14.967954 16394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:42:14.968034 16394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:14.968152 16394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:14.968235 16394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:42:14.968321 16394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:42:14.978374 16394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:14.979733 16394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:14.981104 16394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:15.003924 16403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:15.003997 16400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:15.003975 16401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:15.004370 16394 server_base.cc:1034] running on GCE node
I20260621 02:42:15.005331 16394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:15.006933 16394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:15.008177 16394 hybrid_clock.cc:648] HybridClock initialized: now 1782009735008113 us; error 39 us; skew 500 ppm
I20260621 02:42:15.008564 16394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:15 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:15.018052 16394 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:42:15.019745 16394 webserver.cc:460] Webserver started at http://127.0.97.62:34877/ using document root <none> and password file <none>
I20260621 02:42:15.020493 16394 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:15.020597 16394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:15.026531 16394 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:42:15.029263 16410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:15.030755 16394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:42:15.030933 16394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "08b9a1b422254be6af9b80a526fb1e5f"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "3839e081361a1da2d6bc6cdb75c76970"
server_key_iv: "54738a225a829b312d013cbb7920ca10"
server_key_version: "encryptionkey@0"
I20260621 02:42:15.031960 16394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:15.047570 16394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:15.052043 16394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:15.052301 16394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:15.071336 16394 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46193
I20260621 02:42:15.071367 16461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46193 every 8 connection(s)
I20260621 02:42:15.072912 16394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:42:15.078091 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 16394
I20260621 02:42:15.078923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44271
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46193
--webserver_port=43487
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:42:15.079890 16462 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:42:15.083127 16462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Bootstrap starting.
I20260621 02:42:15.102411 16462 log.cc:824] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Log is configured to *not* fsync() on all Append() calls
I20260621 02:42:15.114125 16462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:42:15.114691 16462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Bootstrap complete.
I20260621 02:42:15.123991 16462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:42:15.124848 16462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b9a1b422254be6af9b80a526fb1e5f, State: Initialized, Role: FOLLOWER
I20260621 02:42:15.125516 16462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:42:15.125774 16462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:42:15.125906 16462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:42:15.126091 16462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:42:15.129839 16462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:42:15.130357 16462 leader_election.cc:302] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08b9a1b422254be6af9b80a526fb1e5f; no voters:
I20260621 02:42:15.130980 16462 leader_election.cc:288] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:42:15.131119 16466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:42:15.132333 16466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 2 LEADER]: Becoming Leader. State: Replica: 08b9a1b422254be6af9b80a526fb1e5f, State: Running, Role: LEADER
I20260621 02:42:15.132926 16466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:42:15.133157 16462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:42:15.135511 16467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "08b9a1b422254be6af9b80a526fb1e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } } }
I20260621 02:42:15.135951 16467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: This master's current role is: LEADER
I20260621 02:42:15.137300 16468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08b9a1b422254be6af9b80a526fb1e5f. Latest consensus state: current_term: 2 leader_uuid: "08b9a1b422254be6af9b80a526fb1e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } } }
I20260621 02:42:15.137714 16468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: This master's current role is: LEADER
I20260621 02:42:15.138736 16472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:42:15.146657 16472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:42:15.148036 16472 catalog_manager.cc:1259] Loaded cluster ID: 6e2b127ca44a4e8fb74a1f36c10df654
I20260621 02:42:15.148193 16472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:42:15.152097 16472 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:42:15.153687 16472 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Loaded TSK: 0
I20260621 02:42:15.195303 16472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Generated new TSK 1
I20260621 02:42:15.195815 16472 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:42:15.454373 16464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:15.454668 16464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:15.454813 16464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:15.462363 16464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:15.462469 16464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:15.462633 16464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:42:15.472470 16464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43487
--tserver_master_addrs=127.0.97.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:15.473897 16464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:15.475188 16464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:15.493564 16490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:15.493604 16489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:15.494572 16492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:15.495995 16464 server_base.cc:1034] running on GCE node
I20260621 02:42:15.496665 16464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:15.497717 16464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:15.498943 16464 hybrid_clock.cc:648] HybridClock initialized: now 1782009735498882 us; error 56 us; skew 500 ppm
I20260621 02:42:15.499253 16464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:15 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:15.502524 16464 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:42:15.504071 16464 webserver.cc:460] Webserver started at http://127.0.97.1:43487/ using document root <none> and password file <none>
I20260621 02:42:15.504846 16464 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:15.505004 16464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:15.510602 16464 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:42:15.514147 16499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:15.515864 16464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:42:15.516047 16464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "83fb564fab5a4aae87b0aa8a062c1982"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "22380482e646eedcb21f8d741036c084"
server_key_iv: "4974de58057a5708d31b7cfe15943c41"
server_key_version: "encryptionkey@0"
I20260621 02:42:15.517074 16464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:15.543404 16464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:15.547495 16464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:15.547778 16464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:15.548751 16464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:15.550096 16464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:42:15.550189 16464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:15.550336 16464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:15.550429 16464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:15.582782 16464 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44271
I20260621 02:42:15.582895 16611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44271 every 8 connection(s)
I20260621 02:42:15.584250 16464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:42:15.592902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 16464
I20260621 02:42:15.593616 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40965
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46193
--webserver_port=40133
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:42:15 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:42:15.609616 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:15.587126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35295 (local address 127.0.97.62:46193)
0621 02:42:15.587680 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:15.587691 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:15.588389 (+ 698us) server_negotiation.cc:407] Connection header received
0621 02:42:15.590514 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:15.590526 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:15.590820 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:15.591116 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:15.592431 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:15.594033 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:15.596568 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:15.597238 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:15.597460 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:15.601841 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:15.601854 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:15.601873 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:15.601923 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:15.605266 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:15.606020 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:15.606028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:15.606041 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:15.606200 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:15.606615 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:15.606624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:15.606626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:15.607049 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:15.607260 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:15.607969 (+ 709us) server_negotiation.cc:299] Negotiation successful
0621 02:42:15.608346 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":247,"threads_started":1}
I20260621 02:42:15.609615 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:15.587361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:35295)
0621 02:42:15.588225 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:15.588267 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:15.590051 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:15.591238 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:15.591244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:15.591726 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:42:15.592214 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:15.592230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:15.594220 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:15.594228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:15.596400 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:15.596418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:15.597513 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:15.597522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:15.597826 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:15.598603 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:42:15.598638 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:42:15.601539 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:42:15.605458 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:15.605467 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:15.605488 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:15.605885 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:15.606326 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:15.606330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:15.606332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:15.606494 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:15.607197 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:42:15.607205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:42:15.607507 (+ 302us) client_negotiation.cc:769] Sending connection context
0621 02:42:15.607782 (+ 275us) client_negotiation.cc:240] Negotiation successful
0621 02:42:15.608295 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":367,"threads_started":1}
I20260621 02:42:15.612495 16612 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46193
I20260621 02:42:15.612882 16612 heartbeater.cc:464] Registering TS with master...
I20260621 02:42:15.613859 16612 heartbeater.cc:511] Master 127.0.97.62:46193 requested a full tablet report, sending...
I20260621 02:42:15.616132 16427 ts_manager.cc:194] Registered new tserver with Master: 83fb564fab5a4aae87b0aa8a062c1982 (127.0.97.1:44271)
I20260621 02:42:15.617645 16427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35295
W20260621 02:42:15.914688 16616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:15.914958 16616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:15.915105 16616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:15.922173 16616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:15.922290 16616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:15.922480 16616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:42:15.931633 16616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40133
--tserver_master_addrs=127.0.97.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:15.932824 16616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:15.934166 16616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:15.946882 16622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:15.946909 16621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:15.946838 16624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:15.947624 16616 server_base.cc:1034] running on GCE node
I20260621 02:42:15.948201 16616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:15.949033 16616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:15.950248 16616 hybrid_clock.cc:648] HybridClock initialized: now 1782009735950196 us; error 71 us; skew 500 ppm
I20260621 02:42:15.950534 16616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:15 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:15.953426 16616 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:42:15.954789 16616 webserver.cc:460] Webserver started at http://127.0.97.2:40133/ using document root <none> and password file <none>
I20260621 02:42:15.955356 16616 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:15.955463 16616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:15.960352 16616 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:42:15.962903 16631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:15.964147 16616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:42:15.964334 16616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "530274f8484b4b4aaa55bc4015b2f5ba"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "7ee98eee0f3e62a0f6ecfe6d9755ed61"
server_key_iv: "098c64cd9ba51adc0fb0e72ca9c9d64c"
server_key_version: "encryptionkey@0"
I20260621 02:42:15.965219 16616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:16.001091 16616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:16.004992 16616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:16.005260 16616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:16.006289 16616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:16.007663 16616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:42:16.007766 16616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:16.007920 16616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:16.008006 16616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:16.041399 16616 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40965
I20260621 02:42:16.041527 16743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40965 every 8 connection(s)
I20260621 02:42:16.042798 16616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:42:16.048729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 16616
I20260621 02:42:16.049459 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46117
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46193
--webserver_port=44547
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:42:16 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:42:16.062597 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.045345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43305 (local address 127.0.97.62:46193)
0621 02:42:16.045515 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.045520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.046696 (+ 1176us) server_negotiation.cc:407] Connection header received
0621 02:42:16.048383 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.048389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.048478 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.048601 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:16.050160 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.051319 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.052820 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.053608 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.053745 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.057183 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:16.057187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:16.057191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:16.057221 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:16.059714 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:16.060403 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:16.060411 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:16.060416 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:16.060539 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:16.060950 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:16.060957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:16.060960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:16.061216 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:16.061397 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.062034 (+ 637us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.062254 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:42:16.063537 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.045831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:43305)
0621 02:42:16.046538 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:16.046574 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:16.048132 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:16.048778 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:16.048786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:16.049389 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:42:16.049992 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.050009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.051494 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.051498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.052671 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.052688 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.053830 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.053836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.054013 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.054677 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:42:16.054702 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:42:16.056960 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:42:16.059851 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:16.059866 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:16.059882 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:16.060281 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:16.060648 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:16.060653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:16.060657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:16.060832 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:16.061382 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:42:16.061390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:42:16.061700 (+ 310us) client_negotiation.cc:769] Sending connection context
0621 02:42:16.061964 (+ 264us) client_negotiation.cc:240] Negotiation successful
0621 02:42:16.062453 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":293,"threads_started":1}
I20260621 02:42:16.064935 16744 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46193
I20260621 02:42:16.065244 16744 heartbeater.cc:464] Registering TS with master...
I20260621 02:42:16.066114 16744 heartbeater.cc:511] Master 127.0.97.62:46193 requested a full tablet report, sending...
I20260621 02:42:16.067708 16427 ts_manager.cc:194] Registered new tserver with Master: 530274f8484b4b4aaa55bc4015b2f5ba (127.0.97.2:40965)
I20260621 02:42:16.068606 16427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43305
W20260621 02:42:16.366655 16747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:16.366959 16747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:16.367123 16747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:16.374617 16747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:16.374747 16747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:16.374969 16747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:42:16.385051 16747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44547
--tserver_master_addrs=127.0.97.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:16.386472 16747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:16.387804 16747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:16.401158 16752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:16.401198 16755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:16.401118 16753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:16.402441 16747 server_base.cc:1034] running on GCE node
I20260621 02:42:16.403087 16747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:16.404099 16747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:16.405342 16747 hybrid_clock.cc:648] HybridClock initialized: now 1782009736405235 us; error 62 us; skew 500 ppm
I20260621 02:42:16.405671 16747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:16 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:16.408877 16747 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:42:16.410519 16747 webserver.cc:460] Webserver started at http://127.0.97.3:44547/ using document root <none> and password file <none>
I20260621 02:42:16.411232 16747 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:16.411397 16747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:16.416755 16747 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:42:16.419591 16762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:16.420953 16747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:42:16.421133 16747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "9553268e49b945f1a2f5e2b1af147775"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "af5ed90de37367e1a42a7eac9c3e4f99"
server_key_iv: "1e541388d21c46ede62c0ae9d5709005"
server_key_version: "encryptionkey@0"
I20260621 02:42:16.422214 16747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:16.448352 16747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:16.452723 16747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:16.453037 16747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:16.454067 16747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:16.455343 16747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:42:16.455472 16747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:16.455595 16747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:16.455679 16747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:16.488327 16747 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46117
I20260621 02:42:16.488413 16874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46117 every 8 connection(s)
I20260621 02:42:16.489869 16747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:42:16.493853 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 16747
Jun 21 02:42:16 dist-test-slave-jkn7 krb5kdc[15843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:42:16.510052 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.494851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.0.97.62:46193)
0621 02:42:16.495604 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.495610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.495629 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:42:16.495692 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.495697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.495772 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.495887 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:16.497409 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.498478 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.500252 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.500911 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.501047 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.503969 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:16.503976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:16.503981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:16.504021 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:16.507140 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:16.507906 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:16.507910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:16.507912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:16.508034 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:16.508452 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:16.508461 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:16.508466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:16.508772 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:16.509049 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.509376 (+ 327us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.509611 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":539,"threads_started":1}
I20260621 02:42:16.512374 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.492443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46801 (local address 127.0.97.62:46193)
0621 02:42:16.492661 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.492667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.493917 (+ 1250us) server_negotiation.cc:407] Connection header received
0621 02:42:16.496442 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.496448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.496519 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.496627 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:16.498453 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.499613 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.501695 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.502348 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.502487 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.506278 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:16.506289 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:16.506292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:16.506333 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:16.509083 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:16.509961 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:16.509968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:16.509974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:16.510130 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:16.510601 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:16.510606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:16.510609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:16.510940 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:16.511097 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.511809 (+ 712us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.512014 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:42:16.513407 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.492933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:46801)
0621 02:42:16.493738 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:16.493790 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:16.496197 (+ 2407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:16.496856 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:16.496864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:16.497437 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:42:16.498194 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.498216 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.499787 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.499792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.501530 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.501562 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.502567 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.502581 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.502767 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.503386 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:42:16.503408 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:42:16.506060 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:42:16.509225 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:16.509234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:16.509254 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:16.509748 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:16.510273 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:16.510278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:16.510282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:16.510475 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:16.511078 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:42:16.511087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:42:16.511437 (+ 350us) client_negotiation.cc:769] Sending connection context
0621 02:42:16.511699 (+ 262us) client_negotiation.cc:240] Negotiation successful
0621 02:42:16.512254 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":378,"threads_started":1}
I20260621 02:42:16.514709 16875 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46193
I20260621 02:42:16.515027 16875 heartbeater.cc:464] Registering TS with master...
I20260621 02:42:16.515872 16875 heartbeater.cc:511] Master 127.0.97.62:46193 requested a full tablet report, sending...
I20260621 02:42:16.517640 16427 ts_manager.cc:194] Registered new tserver with Master: 9553268e49b945f1a2f5e2b1af147775 (127.0.97.3:46117)
I20260621 02:42:16.518863 16427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46801
I20260621 02:42:16.524691 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:42:16.530789 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.525623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.0.97.62:46193)
0621 02:42:16.525774 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.525779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.526196 (+ 417us) server_negotiation.cc:407] Connection header received
0621 02:42:16.526394 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.526398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.526528 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.526642 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:42:16.527054 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.527766 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.529190 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.529653 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.529751 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.530195 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:42:16.530360 (+ 165us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0621 02:42:16.530619 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45314: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260621 02:42:16.530911 16614 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45314: token expired
W20260621 02:42:16.531420 16348 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:46193)
I20260621 02:42:16.544061 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.531859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.0.97.62:46193)
0621 02:42:16.532012 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.532019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.532040 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:42:16.532210 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.532215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.532287 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.532394 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:16.533734 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.534522 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.535929 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.536314 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.536465 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.538919 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:16.538925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:16.538929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:16.538966 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:16.541565 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:16.542215 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:16.542221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:16.542224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:16.542347 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:16.542717 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:16.542722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:16.542725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:16.542999 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:16.543260 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.543546 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.543772 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:42:16.547174 16427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:42:16.554342 16427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:42:16.575453 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.566085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.0.97.1:44271)
0621 02:42:16.566670 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.566679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.567593 (+ 914us) server_negotiation.cc:407] Connection header received
0621 02:42:16.567890 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.567901 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.568169 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.568366 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:16.569265 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.570827 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.572828 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.574281 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.574409 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.574607 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.574961 (+ 354us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.575126 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":302,"threads_started":1}
I20260621 02:42:16.575559 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.565914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:56922)
0621 02:42:16.567442 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:16.567479 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:16.567671 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:16.568434 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:16.568442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:16.568475 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:16.569071 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.569089 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.570952 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.570959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.572623 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.572647 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.574514 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.574524 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.574745 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.574796 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 02:42:16.574914 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 02:42:16.575096 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"mutex_wait_us":5,"thread_start_us":1149,"threads_started":1}
I20260621 02:42:16.577544 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.566009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40965 (local address 127.0.0.1:57816)
0621 02:42:16.568224 (+ 2215us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:16.568241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:16.568369 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:16.570122 (+ 1753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:16.570127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:16.570151 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:16.570501 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.570514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.572437 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.572445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.574651 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.574671 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.576684 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.576689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.576937 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.576965 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 02:42:16.577032 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 02:42:16.577126 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2103,"mutex_wait_us":2,"thread_start_us":1984,"threads_started":1}
I20260621 02:42:16.577566 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.568638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46117 (local address 127.0.0.1:53378)
0621 02:42:16.569405 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:16.569421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:16.569526 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:16.570387 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:16.570393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:16.570423 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:16.570812 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.570824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.573103 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.573108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.574750 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.574763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.576559 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.576565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.576813 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.576849 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 02:42:16.576944 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:42:16.577074 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":56,"thread_start_us":450,"threads_started":1}
I20260621 02:42:16.578013 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.568349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.0.97.3:46117)
0621 02:42:16.569159 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.569168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.569631 (+ 463us) server_negotiation.cc:407] Connection header received
0621 02:42:16.569725 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.569731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.569930 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.570115 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:16.571017 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.572939 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.574877 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.576231 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.576357 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.576561 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.577364 (+ 803us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.577527 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":421,"threads_started":1}
I20260621 02:42:16.578244 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.566663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.97.2:40965)
0621 02:42:16.567283 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.567293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.569325 (+ 2032us) server_negotiation.cc:407] Connection header received
0621 02:42:16.569513 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.569522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.569750 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.570133 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:16.570681 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.572278 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.574802 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.576230 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.576361 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.576535 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.577818 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.577970 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":370,"threads_started":1}
I20260621 02:42:16.580179 16547 tablet_service.cc:1467] Processing CreateTablet for tablet 0b10d5346384446597c24a71598b0cfd (DEFAULT_TABLE table=test-table [id=06f46721f86048d79dd65eb575122a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:42:16.580855 16679 tablet_service.cc:1467] Processing CreateTablet for tablet 0b10d5346384446597c24a71598b0cfd (DEFAULT_TABLE table=test-table [id=06f46721f86048d79dd65eb575122a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:42:16.581636 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b10d5346384446597c24a71598b0cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:42:16.581398 16810 tablet_service.cc:1467] Processing CreateTablet for tablet 0b10d5346384446597c24a71598b0cfd (DEFAULT_TABLE table=test-table [id=06f46721f86048d79dd65eb575122a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:42:16.581960 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b10d5346384446597c24a71598b0cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:42:16.582585 16810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b10d5346384446597c24a71598b0cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:42:16.596346 16891 tablet_bootstrap.cc:492] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: Bootstrap starting.
I20260621 02:42:16.598135 16892 tablet_bootstrap.cc:492] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: Bootstrap starting.
I20260621 02:42:16.600482 16891 tablet_bootstrap.cc:654] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: Neither blocks nor log segments found. Creating new log.
I20260621 02:42:16.601504 16893 tablet_bootstrap.cc:492] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Bootstrap starting.
I20260621 02:42:16.602128 16892 tablet_bootstrap.cc:654] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: Neither blocks nor log segments found. Creating new log.
I20260621 02:42:16.602382 16891 log.cc:824] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: Log is configured to *not* fsync() on all Append() calls
I20260621 02:42:16.603260 16892 log.cc:824] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: Log is configured to *not* fsync() on all Append() calls
I20260621 02:42:16.604784 16893 tablet_bootstrap.cc:654] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Neither blocks nor log segments found. Creating new log.
I20260621 02:42:16.605665 16891 tablet_bootstrap.cc:492] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: No bootstrap required, opened a new log
I20260621 02:42:16.605960 16891 ts_tablet_manager.cc:1397] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260621 02:42:16.606051 16893 log.cc:824] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Log is configured to *not* fsync() on all Append() calls
I20260621 02:42:16.607817 16892 tablet_bootstrap.cc:492] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: No bootstrap required, opened a new log
I20260621 02:42:16.608080 16892 ts_tablet_manager.cc:1397] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:42:16.608943 16893 tablet_bootstrap.cc:492] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: No bootstrap required, opened a new log
I20260621 02:42:16.609203 16893 ts_tablet_manager.cc:1397] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:42:16.615721 16892 raft_consensus.cc:348] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } }
I20260621 02:42:16.616142 16892 raft_consensus.cc:374] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:42:16.616246 16892 raft_consensus.cc:729] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553268e49b945f1a2f5e2b1af147775, State: Initialized, Role: FOLLOWER
I20260621 02:42:16.616911 16892 consensus_queue.cc:260] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } }
I20260621 02:42:16.616854 16891 raft_consensus.cc:348] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } }
I20260621 02:42:16.617301 16891 raft_consensus.cc:374] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:42:16.617406 16891 raft_consensus.cc:729] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530274f8484b4b4aaa55bc4015b2f5ba, State: Initialized, Role: FOLLOWER
I20260621 02:42:16.618145 16875 heartbeater.cc:503] Master 127.0.97.62:46193 was elected leader, sending a full tablet report...
I20260621 02:42:16.618108 16891 consensus_queue.cc:260] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } }
I20260621 02:42:16.618975 16892 ts_tablet_manager.cc:1428] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260621 02:42:16.620568 16893 raft_consensus.cc:348] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } }
I20260621 02:42:16.621129 16612 heartbeater.cc:503] Master 127.0.97.62:46193 was elected leader, sending a full tablet report...
I20260621 02:42:16.621312 16893 raft_consensus.cc:374] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:42:16.621428 16893 raft_consensus.cc:729] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fb564fab5a4aae87b0aa8a062c1982, State: Initialized, Role: FOLLOWER
I20260621 02:42:16.622062 16893 consensus_queue.cc:260] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } }
I20260621 02:42:16.625806 16744 heartbeater.cc:503] Master 127.0.97.62:46193 was elected leader, sending a full tablet report...
I20260621 02:42:16.626926 16891 ts_tablet_manager.cc:1428] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260621 02:42:16.627267 16893 ts_tablet_manager.cc:1428] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
W20260621 02:42:16.741456 16876 tablet.cc:2369] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:42:16.794842 16745 tablet.cc:2369] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:42:16.836524 16613 tablet.cc:2369] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:42:16.848855 16899 raft_consensus.cc:482] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:42:16.849138 16899 raft_consensus.cc:504] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } }
I20260621 02:42:16.850879 16899 leader_election.cc:288] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530274f8484b4b4aaa55bc4015b2f5ba (127.0.97.2:40965), 9553268e49b945f1a2f5e2b1af147775 (127.0.97.3:46117)
I20260621 02:42:16.857535 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.851286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59287 (local address 127.0.97.2:40965)
0621 02:42:16.851507 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.851515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.852391 (+ 876us) server_negotiation.cc:407] Connection header received
0621 02:42:16.852625 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.852629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.852685 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.852780 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:16.853406 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.854110 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.855864 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.856599 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.856758 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.856862 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.857141 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.857262 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:42:16.857647 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.851291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51545 (local address 127.0.97.3:46117)
0621 02:42:16.851519 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.851526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.852486 (+ 960us) server_negotiation.cc:407] Connection header received
0621 02:42:16.852677 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.852691 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.852760 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.852856 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:16.853549 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.854247 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.855981 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.856752 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.856896 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.857003 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.857261 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.857403 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:42:16.857707 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.851421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40965 (local address 127.0.97.1:59287)
0621 02:42:16.852270 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:16.852286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:16.852477 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:16.852793 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:16.852798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:16.852822 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:16.853237 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.853247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.854245 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.854252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.855701 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.855715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.856810 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.856817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.856986 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.857009 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:42:16.857114 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 02:42:16.857236 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":4,"spinlock_wait_cycles":25728,"thread_start_us":562,"threads_started":1}
I20260621 02:42:16.857806 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.851423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46117 (local address 127.0.97.1:51545)
0621 02:42:16.852377 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:16.852391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:16.852543 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:16.852868 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:16.852876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:16.852925 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:16.853384 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.853394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.854385 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.854391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.855825 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:16.855837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.856968 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.856974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:16.857114 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.857135 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:42:16.857236 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 02:42:16.857372 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":24,"spinlock_wait_cycles":29184,"thread_start_us":469,"threads_started":1}
I20260621 02:42:16.858650 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b10d5346384446597c24a71598b0cfd" candidate_uuid: "83fb564fab5a4aae87b0aa8a062c1982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" is_pre_election: true
I20260621 02:42:16.858688 16830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b10d5346384446597c24a71598b0cfd" candidate_uuid: "83fb564fab5a4aae87b0aa8a062c1982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553268e49b945f1a2f5e2b1af147775" is_pre_election: true
I20260621 02:42:16.859108 16699 raft_consensus.cc:2454] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83fb564fab5a4aae87b0aa8a062c1982 in term 0.
I20260621 02:42:16.859133 16830 raft_consensus.cc:2454] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83fb564fab5a4aae87b0aa8a062c1982 in term 0.
I20260621 02:42:16.859697 16502 leader_election.cc:302] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530274f8484b4b4aaa55bc4015b2f5ba, 83fb564fab5a4aae87b0aa8a062c1982; no voters:
I20260621 02:42:16.860055 16899 raft_consensus.cc:2781] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:42:16.860201 16899 raft_consensus.cc:482] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:42:16.860306 16899 raft_consensus.cc:3037] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:42:16.862643 16899 raft_consensus.cc:504] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } }
I20260621 02:42:16.863260 16899 leader_election.cc:288] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [CANDIDATE]: Term 1 election: Requested vote from peers 530274f8484b4b4aaa55bc4015b2f5ba (127.0.97.2:40965), 9553268e49b945f1a2f5e2b1af147775 (127.0.97.3:46117)
I20260621 02:42:16.863644 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b10d5346384446597c24a71598b0cfd" candidate_uuid: "83fb564fab5a4aae87b0aa8a062c1982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530274f8484b4b4aaa55bc4015b2f5ba"
I20260621 02:42:16.863770 16830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b10d5346384446597c24a71598b0cfd" candidate_uuid: "83fb564fab5a4aae87b0aa8a062c1982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553268e49b945f1a2f5e2b1af147775"
I20260621 02:42:16.863864 16699 raft_consensus.cc:3037] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:42:16.863960 16830 raft_consensus.cc:3037] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:42:16.867041 16699 raft_consensus.cc:2454] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83fb564fab5a4aae87b0aa8a062c1982 in term 1.
I20260621 02:42:16.867041 16830 raft_consensus.cc:2454] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83fb564fab5a4aae87b0aa8a062c1982 in term 1.
I20260621 02:42:16.867436 16502 leader_election.cc:302] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530274f8484b4b4aaa55bc4015b2f5ba, 83fb564fab5a4aae87b0aa8a062c1982; no voters:
I20260621 02:42:16.867720 16899 raft_consensus.cc:2781] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:42:16.868134 16899 raft_consensus.cc:686] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 1 LEADER]: Becoming Leader. State: Replica: 83fb564fab5a4aae87b0aa8a062c1982, State: Running, Role: LEADER
I20260621 02:42:16.868657 16899 consensus_queue.cc:237] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } }
I20260621 02:42:16.873337 16427 catalog_manager.cc:5495] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83fb564fab5a4aae87b0aa8a062c1982 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "83fb564fab5a4aae87b0aa8a062c1982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb564fab5a4aae87b0aa8a062c1982" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 } health_report { overall_health: UNKNOWN } } }
I20260621 02:42:16.945649 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.940159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.0.97.1:44271)
0621 02:42:16.940374 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.940380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.940496 (+ 116us) server_negotiation.cc:407] Connection header received
0621 02:42:16.940757 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.940762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.940849 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.941036 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:42:16.941491 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.942211 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.943506 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.943927 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.944061 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:16.944557 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:42:16.945051 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:42:16.945215 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:16.945271 (+ 56us) server_negotiation.cc:299] Negotiation successful
0621 02:42:16.945366 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:42:16.953558 16699 raft_consensus.cc:1261] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [term 1 FOLLOWER]: Refusing update from remote peer 83fb564fab5a4aae87b0aa8a062c1982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:42:16.953573 16830 raft_consensus.cc:1261] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [term 1 FOLLOWER]: Refusing update from remote peer 83fb564fab5a4aae87b0aa8a062c1982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:42:16.954581 16902 consensus_queue.cc:1035] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530274f8484b4b4aaa55bc4015b2f5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:42:16.954957 16899 consensus_queue.cc:1035] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553268e49b945f1a2f5e2b1af147775" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:42:16.966063 16905 mvcc.cc:205] Tried to move back new op lower bound from 7299111882551799808 to 7299111882217988096. Current Snapshot: MvccSnapshot[applied={T|T < 7299111882551799808}]
I20260621 02:42:16.979864 16425 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:42:16.980136 16425 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:42:16.983739 16425 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Sending DeleteTablet for 3 replicas of tablet 0b10d5346384446597c24a71598b0cfd
I20260621 02:42:16.984961 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b10d5346384446597c24a71598b0cfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:42:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56922
I20260621 02:42:16.984946 16679 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b10d5346384446597c24a71598b0cfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:42:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57816
I20260621 02:42:16.985482 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 16464
I20260621 02:42:16.985697 16810 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b10d5346384446597c24a71598b0cfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:42:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53378
I20260621 02:42:16.995925 16916 tablet_replica.cc:330] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: stopping tablet replica
I20260621 02:42:16.996529 16916 raft_consensus.cc:2229] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:42:16.997146 16916 raft_consensus.cc:2258] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:42:16.998001 16915 tablet_replica.cc:330] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: stopping tablet replica
I20260621 02:42:16.998431 16915 raft_consensus.cc:2229] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:42:16.999112 16915 raft_consensus.cc:2258] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:42:16.999886 16917 tablet_replica.cc:330] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: stopping tablet replica
I20260621 02:42:17.000252 16917 raft_consensus.cc:2229] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:42:17.000731 16917 raft_consensus.cc:2258] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:42:17.000851 16916 ts_tablet_manager.cc:1905] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:42:17.004760 16917 ts_tablet_manager.cc:1905] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:42:17.004796 16915 ts_tablet_manager.cc:1905] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:42:17.008644 16916 ts_tablet_manager.cc:1918] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:42:17.008802 16916 log.cc:1196] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/0b10d5346384446597c24a71598b0cfd
I20260621 02:42:17.009343 16916 ts_tablet_manager.cc:1939] T 0b10d5346384446597c24a71598b0cfd P 530274f8484b4b4aaa55bc4015b2f5ba: Deleting consensus metadata
I20260621 02:42:17.010491 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:16.987268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.0.97.1:44271)
0621 02:42:16.987482 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:16.987491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:16.987513 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:42:16.987580 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:16.987588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:16.987679 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:16.987792 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:16.990624 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.992261 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.994683 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:16.995561 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:16.995682 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:17.003091 (+ 7409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:17.003104 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:17.003120 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:17.003172 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:17.007206 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:17.008020 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:17.008027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:17.008031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:17.008201 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:17.008650 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:17.008658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:17.008661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:17.009228 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:17.009709 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:17.009881 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 02:42:17.010109 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:42:17.011493 16607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:17.011475 16413 catalog_manager.cc:4841] TS 530274f8484b4b4aaa55bc4015b2f5ba (127.0.97.2:40965): tablet 0b10d5346384446597c24a71598b0cfd (table test-table [id=06f46721f86048d79dd65eb575122a4f]) successfully deleted
I20260621 02:42:17.015301 16917 ts_tablet_manager.cc:1918] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:42:17.015479 16917 log.cc:1196] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/0b10d5346384446597c24a71598b0cfd
I20260621 02:42:17.015919 16917 ts_tablet_manager.cc:1939] T 0b10d5346384446597c24a71598b0cfd P 9553268e49b945f1a2f5e2b1af147775: Deleting consensus metadata
I20260621 02:42:17.017416 16414 catalog_manager.cc:4841] TS 9553268e49b945f1a2f5e2b1af147775 (127.0.97.3:46117): tablet 0b10d5346384446597c24a71598b0cfd (table test-table [id=06f46721f86048d79dd65eb575122a4f]) successfully deleted
I20260621 02:42:17.106897 16915 ts_tablet_manager.cc:1918] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:42:17.107199 16915 log.cc:1196] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/0b10d5346384446597c24a71598b0cfd
I20260621 02:42:17.107631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 16464
W20260621 02:42:17.125762 16414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44271 (error 108)
I20260621 02:42:17.126075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 16616
I20260621 02:42:17.127764 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:17.127059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:56952)
0621 02:42:17.127514 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:17.127614 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":233,"threads_started":1}
W20260621 02:42:17.128027 16414 catalog_manager.cc:4584] TS 83fb564fab5a4aae87b0aa8a062c1982 (127.0.97.1:44271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b10d5346384446597c24a71598b0cfd: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
I20260621 02:42:17.140292 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:17.126915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.97.2:40965)
0621 02:42:17.127085 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:17.127091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:17.127141 (+ 50us) server_negotiation.cc:407] Connection header received
0621 02:42:17.127412 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:17.127420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:17.127528 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:17.127629 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:17.129046 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:17.129912 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:17.131252 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:17.131647 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:17.131780 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:17.134386 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:17.134398 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:17.134412 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:17.134448 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:17.137597 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:17.138252 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:17.138259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:17.138261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:17.138393 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:17.138815 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:17.138819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:17.138821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:17.139169 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:17.139406 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:17.139776 (+ 370us) server_negotiation.cc:299] Negotiation successful
0621 02:42:17.139936 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:42:17.140980 16739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:17.181854 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:17.181402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:56968)
0621 02:42:17.181621 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:17.181713 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 02:42:17.182989 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:17.182612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:56984)
0621 02:42:17.182785 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:17.182858 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 02:42:17.230108 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 16616
I20260621 02:42:17.243618 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:17.243176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:56988)
0621 02:42:17.243383 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:17.243469 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 02:42:17.244661 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:17.244319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:56994)
0621 02:42:17.244468 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:17.244532 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:42:17.248202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 16747
I20260621 02:42:17.260972 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:17.248946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.0.97.3:46117)
0621 02:42:17.249113 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:17.249118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:17.249137 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:42:17.249343 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:17.249347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:17.249428 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:17.249517 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:17.250729 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:17.251413 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:17.252718 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:17.253102 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:17.253206 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:17.255900 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:17.255913 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:17.255930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:17.255959 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:17.258623 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:17.259273 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:17.259281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:17.259283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:17.259424 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:17.259749 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:17.259752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:17.259754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:17.260094 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:17.260306 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:17.260557 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 02:42:17.260712 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:42:17.261732 16870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:17.325110 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:17.324619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:57006)
0621 02:42:17.324864 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:17.324961 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 02:42:17.326316 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:17.325901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:57012)
0621 02:42:17.326118 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:17.326186 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260621 02:42:17.349506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 16747
I20260621 02:42:17.368021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 16394
I20260621 02:42:17.369259 16457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:17.485659 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 16394
I20260621 02:42:37.508242 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46193
--webserver_interface=127.0.97.62
--webserver_port=34877
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:42:37.833194 16924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:37.833612 16924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:37.833787 16924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:37.840948 16924 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 02:42:37.841039 16924 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 02:42:37.841110 16924 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 02:42:37.841187 16924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:42:37.841269 16924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:37.841403 16924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:37.841481 16924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:42:37.841593 16924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:42:37.851279 16924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:37.852522 16924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:37.853864 16924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:37.865335 16932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:37.865391 16929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:37.865371 16930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:37.866163 16924 server_base.cc:1034] running on GCE node
I20260621 02:42:37.866919 16924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:37.868279 16924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:37.869518 16924 hybrid_clock.cc:648] HybridClock initialized: now 1782009757869463 us; error 33 us; skew 500 ppm
I20260621 02:42:37.869860 16924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:37 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:37.872957 16924 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:42:37.874500 16924 webserver.cc:460] Webserver started at http://127.0.97.62:34877/ using document root <none> and password file <none>
I20260621 02:42:37.875195 16924 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:37.875329 16924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:37.880716 16924 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:42:37.883586 16939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:37.884974 16924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:42:37.885172 16924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "08b9a1b422254be6af9b80a526fb1e5f"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "3839e081361a1da2d6bc6cdb75c76970"
server_key_iv: "54738a225a829b312d013cbb7920ca10"
server_key_version: "encryptionkey@0"
I20260621 02:42:37.886199 16924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:37.900933 16924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:37.905437 16924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:37.905730 16924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:37.924319 16924 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46193
I20260621 02:42:37.924402 16990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46193 every 8 connection(s)
I20260621 02:42:37.925844 16924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:42:37.931118 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 16924
I20260621 02:42:37.932086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44271
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46193
--webserver_port=43487
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:42:37.933188 16991 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:42:37.938226 16991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Bootstrap starting.
I20260621 02:42:37.958966 16991 log.cc:824] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Log is configured to *not* fsync() on all Append() calls
I20260621 02:42:37.974902 16991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:42:37.975442 16991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Bootstrap complete.
I20260621 02:42:37.984306 16991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:42:37.985185 16991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b9a1b422254be6af9b80a526fb1e5f, State: Initialized, Role: FOLLOWER
I20260621 02:42:37.985901 16991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:42:37.986131 16991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:42:37.986250 16991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:42:37.986441 16991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 2 FOLLOWER]: Advancing to term 3
I20260621 02:42:37.990401 16991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:42:37.990924 16991 leader_election.cc:302] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08b9a1b422254be6af9b80a526fb1e5f; no voters:
I20260621 02:42:37.991659 16991 leader_election.cc:288] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 02:42:37.991807 16995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 3 FOLLOWER]: Leader election won for term 3
I20260621 02:42:37.993564 16995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [term 3 LEADER]: Becoming Leader. State: Replica: 08b9a1b422254be6af9b80a526fb1e5f, State: Running, Role: LEADER
I20260621 02:42:37.993964 16991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:42:37.994120 16995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } }
I20260621 02:42:37.998942 16997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08b9a1b422254be6af9b80a526fb1e5f. Latest consensus state: current_term: 3 leader_uuid: "08b9a1b422254be6af9b80a526fb1e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } } }
I20260621 02:42:37.999377 16997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: This master's current role is: LEADER
I20260621 02:42:37.999372 16996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "08b9a1b422254be6af9b80a526fb1e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b9a1b422254be6af9b80a526fb1e5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46193 } } }
I20260621 02:42:38.000136 16996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f [sys.catalog]: This master's current role is: LEADER
I20260621 02:42:38.005981 17005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:42:38.017376 17005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:42:38.019567 17005 catalog_manager.cc:1259] Loaded cluster ID: 6e2b127ca44a4e8fb74a1f36c10df654
I20260621 02:42:38.019690 17005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:42:38.025097 17005 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:42:38.027493 17005 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Loaded TSK: 0
I20260621 02:42:38.027657 17005 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08b9a1b422254be6af9b80a526fb1e5f: Loaded TSK: 1
F20260621 02:42:38.029026 17005 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f042ea289ad google::LogMessage::Fail() at ??:0
@ 0x7f042ea2b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f042ea284de google::LogMessage::Flush() at ??:0
@ 0x7f042ea29a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f043278c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f043277b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f04444571c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0444456783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f04444e22a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f04444e20c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f044452c5e9 std::function<>::operator()() at ??:0
@ 0x7f044445acdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0444458c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f04444e0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f04444e0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0431674345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f043167791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0431677799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0431652eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f043492c6db start_thread at ??:0
@ 0x7f042d8c261f clone at ??:0
W20260621 02:42:38.338044 16993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:38.338311 16993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:38.338407 16993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:38.345636 16993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:38.345741 16993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:38.345851 16993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:42:38.355546 16993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43487
--tserver_master_addrs=127.0.97.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:38.356832 16993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:38.358088 16993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:38.370821 17038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:38.370931 17041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:38.370965 17039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:38.372004 16993 server_base.cc:1034] running on GCE node
I20260621 02:42:38.372644 16993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:38.373703 16993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:38.374938 16993 hybrid_clock.cc:648] HybridClock initialized: now 1782009758374898 us; error 61 us; skew 500 ppm
I20260621 02:42:38.375272 16993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:38 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:38.378436 16993 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:42:38.379842 16993 webserver.cc:460] Webserver started at http://127.0.97.1:43487/ using document root <none> and password file <none>
I20260621 02:42:38.380518 16993 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:38.380666 16993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:38.386019 16993 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:42:38.390307 17048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:38.391788 16993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:42:38.391985 16993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "83fb564fab5a4aae87b0aa8a062c1982"
format_stamp: "Formatted at 2026-06-21 02:41:52 on dist-test-slave-jkn7"
server_key: "22380482e646eedcb21f8d741036c084"
server_key_iv: "4974de58057a5708d31b7cfe15943c41"
server_key_version: "encryptionkey@0"
I20260621 02:42:38.392997 16993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:38.405781 16993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:38.410125 16993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:38.410389 16993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:38.411386 16993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:38.413223 17055 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260621 02:42:38.418445 17055 ts_tablet_manager.cc:1852] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260621 02:42:38.419487 17055 ts_tablet_manager.cc:1905] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:42:38.426522 17055 ts_tablet_manager.cc:1918] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:42:38.426800 17055 ts_tablet_manager.cc:1939] T 0b10d5346384446597c24a71598b0cfd P 83fb564fab5a4aae87b0aa8a062c1982: Deleting consensus metadata
I20260621 02:42:38.427245 16993 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260621 02:42:38.427386 16993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.001s
I20260621 02:42:38.427526 16993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:38.427624 16993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:38.461531 16993 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44271
I20260621 02:42:38.461588 17161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44271 every 8 connection(s)
I20260621 02:42:38.462839 16993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:42:38.467434 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:38.465758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:56403)
0621 02:42:38.466474 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:38.466643 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":292,"threads_started":1}
W20260621 02:42:38.468209 17162 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46193 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46193: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
I20260621 02:42:38.468564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 16993
I20260621 02:42:38.469146 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:38.468738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:59077)
0621 02:42:38.468911 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:38.469003 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260621 02:42:38.469437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40965
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46193
--webserver_port=40133
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:42:38.470129 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:38.469804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:46565)
0621 02:42:38.469937 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:38.470013 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:42:38.470410 17162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:42:38.796923 17165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:38.797261 17165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:38.797477 17165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:38.804616 17165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:38.804728 17165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:38.804973 17165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:42:38.814683 17165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40133
--tserver_master_addrs=127.0.97.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:38.816016 17165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:38.817313 17165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:38.830317 17170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:38.830356 17173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:38.830396 17171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:38.831121 17165 server_base.cc:1034] running on GCE node
I20260621 02:42:38.831655 17165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:38.832494 17165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:38.833722 17165 hybrid_clock.cc:648] HybridClock initialized: now 1782009758833652 us; error 50 us; skew 500 ppm
I20260621 02:42:38.834072 17165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:38 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:38.837153 17165 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:42:38.838701 17165 webserver.cc:460] Webserver started at http://127.0.97.2:40133/ using document root <none> and password file <none>
I20260621 02:42:38.839327 17165 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:38.839429 17165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:38.844903 17165 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:42:38.847780 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:38.849145 17165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:42:38.849366 17165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "530274f8484b4b4aaa55bc4015b2f5ba"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "7ee98eee0f3e62a0f6ecfe6d9755ed61"
server_key_iv: "098c64cd9ba51adc0fb0e72ca9c9d64c"
server_key_version: "encryptionkey@0"
I20260621 02:42:38.850337 17165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:38.869354 17165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:38.873500 17165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:38.873785 17165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:38.874816 17165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:38.876155 17165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:42:38.876252 17165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:38.876415 17165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:38.876488 17165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:38.910557 17165 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40965
I20260621 02:42:38.910615 17292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40965 every 8 connection(s)
I20260621 02:42:38.911981 17165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:42:38.913182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 17165
I20260621 02:42:38.914000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46117
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46193
--webserver_port=44547
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:42:38.917209 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:38.915225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:43407)
0621 02:42:38.916080 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:38.916314 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":380,"threads_started":1}
W20260621 02:42:38.918004 17293 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46193 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46193: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
I20260621 02:42:38.919018 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:38.918563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:43265)
0621 02:42:38.918760 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:38.918855 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260621 02:42:38.920034 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:38.919602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:44975)
0621 02:42:38.919774 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:38.919874 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:42:38.920333 17293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:42:39.235222 17296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:39.235494 17296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:39.235690 17296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:39.242769 17296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:39.242867 17296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:39.243055 17296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:42:39.252409 17296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44547
--tserver_master_addrs=127.0.97.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:39.253662 17296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:39.254984 17296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:39.267550 17301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:39.267550 17302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:39.267589 17304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:39.268741 17296 server_base.cc:1034] running on GCE node
I20260621 02:42:39.269443 17296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:39.270358 17296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:39.271613 17296 hybrid_clock.cc:648] HybridClock initialized: now 1782009759271575 us; error 59 us; skew 500 ppm
I20260621 02:42:39.271961 17296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:39 dist-test-slave-jkn7 krb5kdc[15843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:39.275146 17296 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:42:39.276646 17296 webserver.cc:460] Webserver started at http://127.0.97.3:44547/ using document root <none> and password file <none>
I20260621 02:42:39.277366 17296 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:39.277511 17296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:39.282691 17296 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:42:39.285540 17311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:39.286919 17296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:42:39.287109 17296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "9553268e49b945f1a2f5e2b1af147775"
format_stamp: "Formatted at 2026-06-21 02:41:53 on dist-test-slave-jkn7"
server_key: "af5ed90de37367e1a42a7eac9c3e4f99"
server_key_iv: "1e541388d21c46ede62c0ae9d5709005"
server_key_version: "encryptionkey@0"
I20260621 02:42:39.288100 17296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:39.311976 17296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:39.316313 17296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:39.316608 17296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:39.317627 17296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:39.319038 17296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:42:39.319159 17296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:39.319290 17296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:39.319383 17296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:39.352871 17296 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46117
I20260621 02:42:39.352960 17423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46117 every 8 connection(s)
I20260621 02:42:39.354923 17296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:42:39.358027 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 17296
I20260621 02:42:39.364327 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:39.359893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:36231)
0621 02:42:39.362811 (+ 2918us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:39.363106 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2581,"thread_start_us":2304,"threads_started":1}
W20260621 02:42:39.364919 17424 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46193 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46193: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
I20260621 02:42:39.365769 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:39.365473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:38295)
0621 02:42:39.365602 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:39.365666 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 02:42:39.366616 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:39.366257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:56309)
0621 02:42:39.366368 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:39.366466 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 02:42:39.366905 17424 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 02:42:39.471722 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:39.470938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:45489)
0621 02:42:39.471411 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:39.471492 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":229,"threads_started":1}
I20260621 02:42:39.921731 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:39.920873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:60637)
0621 02:42:39.921423 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:39.921510 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":225,"threads_started":1}
I20260621 02:42:40.368122 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:40.367372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:34281)
0621 02:42:40.367812 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:40.367893 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":215,"threads_started":1}
I20260621 02:42:40.473435 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:40.472713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:39883)
0621 02:42:40.473146 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:40.473228 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260621 02:42:40.923887 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:40.922606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:43859)
0621 02:42:40.923643 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:40.923732 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":813,"threads_started":1}
I20260621 02:42:41.369746 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:41.368980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:39753)
0621 02:42:41.369498 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:41.369577 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":237,"threads_started":1}
I20260621 02:42:41.475032 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:41.474290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:45045)
0621 02:42:41.474791 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:41.474877 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":279,"threads_started":1}
I20260621 02:42:41.925760 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:41.924837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:33185)
0621 02:42:41.925492 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:41.925581 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":436,"threads_started":1}
I20260621 02:42:42.371397 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:42.370561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:47919)
0621 02:42:42.371021 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:42.371150 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":252,"threads_started":1}
I20260621 02:42:42.476807 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:42.475933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:45837)
0621 02:42:42.476442 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:42.476555 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":251,"threads_started":1}
I20260621 02:42:42.927331 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:42.926599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:51787)
0621 02:42:42.927049 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:42.927137 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260621 02:42:43.373256 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:43.372461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:41161)
0621 02:42:43.372991 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:43.373067 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":297,"threads_started":1}
I20260621 02:42:43.478505 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:43.477848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:55273)
0621 02:42:43.478271 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:43.478351 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":225,"threads_started":1}
I20260621 02:42:43.928949 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:43.928236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:53791)
0621 02:42:43.928707 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:43.928782 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":248,"threads_started":1}
I20260621 02:42:44.374847 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:44.374142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:38987)
0621 02:42:44.374608 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:44.374682 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":254,"threads_started":1}
I20260621 02:42:44.480261 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:44.479449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:48639)
0621 02:42:44.479967 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:44.480067 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":312,"threads_started":1}
I20260621 02:42:44.930512 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:44.929841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:33427)
0621 02:42:44.930257 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:44.930353 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":211,"threads_started":1}
I20260621 02:42:45.376549 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:45.375755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:40753)
0621 02:42:45.376258 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:45.376379 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
I20260621 02:42:45.482123 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:45.481193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:59119)
0621 02:42:45.481832 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:45.481925 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":235,"threads_started":1}
I20260621 02:42:45.932130 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:45.931360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:34263)
0621 02:42:45.931837 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:45.931912 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":217,"threads_started":1}
I20260621 02:42:46.378245 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:46.377457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:35803)
0621 02:42:46.377957 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:46.378034 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":241,"threads_started":1}
I20260621 02:42:46.483733 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:46.483039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:59567)
0621 02:42:46.483483 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:46.483558 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":244,"threads_started":1}
I20260621 02:42:46.933851 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:46.933088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:37411)
0621 02:42:46.933603 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:46.933685 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":289,"threads_started":1}
I20260621 02:42:47.379897 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:47.379206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:43761)
0621 02:42:47.379649 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:47.379732 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":220,"threads_started":1}
I20260621 02:42:47.485627 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:47.484815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:53279)
0621 02:42:47.485360 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:47.485461 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":250,"threads_started":1}
I20260621 02:42:47.935542 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:47.934765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:51747)
0621 02:42:47.935274 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:47.935353 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":240,"threads_started":1}
I20260621 02:42:48.381716 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:48.380873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:37753)
0621 02:42:48.381432 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:48.381530 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":229,"threads_started":1}
I20260621 02:42:48.487280 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:48.486576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:48525)
0621 02:42:48.487020 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:48.487102 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":239,"threads_started":1}
I20260621 02:42:48.937093 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:48.936392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:59357)
0621 02:42:48.936852 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:48.936932 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":261,"threads_started":1}
I20260621 02:42:49.383235 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:49.382562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:34487)
0621 02:42:49.383002 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:49.383078 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":255,"threads_started":1}
I20260621 02:42:49.488859 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:49.488176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:50139)
0621 02:42:49.488620 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:49.488694 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":263,"threads_started":1}
I20260621 02:42:49.938660 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:49.938012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:54003)
0621 02:42:49.938401 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:49.938483 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":195,"threads_started":1}
I20260621 02:42:50.384905 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:50.384169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:39189)
0621 02:42:50.384622 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:50.384710 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":235,"threads_started":1}
I20260621 02:42:50.490483 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:50.489820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:36951)
0621 02:42:50.490229 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:50.490315 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":216,"threads_started":1}
I20260621 02:42:50.940312 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:50.939588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:48823)
0621 02:42:50.940039 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:50.940124 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260621 02:42:51.386819 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:51.385883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:49729)
0621 02:42:51.386450 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:51.386577 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":302,"threads_started":1}
I20260621 02:42:51.492211 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:51.491384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:38045)
0621 02:42:51.491887 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:51.491968 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":255,"threads_started":1}
I20260621 02:42:51.941893 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:51.941154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:59783)
0621 02:42:51.941628 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:51.941707 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
I20260621 02:42:52.388490 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:52.387774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:35913)
0621 02:42:52.388227 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:52.388311 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":208,"threads_started":1}
I20260621 02:42:52.493815 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:52.493025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:36051)
0621 02:42:52.493543 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:52.493626 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":314,"threads_started":1}
I20260621 02:42:52.943521 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:52.942797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:43185)
0621 02:42:52.943256 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:52.943338 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":241,"threads_started":1}
I20260621 02:42:53.390026 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:53.389374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:36581)
0621 02:42:53.389777 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:53.389853 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":204,"threads_started":1}
I20260621 02:42:53.495551 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:53.494817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.1:56789)
0621 02:42:53.495296 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:53.495374 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
I20260621 02:42:53.945468 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:53.944605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.2:44619)
0621 02:42:53.945037 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:53.945132 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":236,"threads_started":1}
I20260621 02:42:54.366549 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 16993
I20260621 02:42:54.388370 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:54.367705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.0.97.1:44271)
0621 02:42:54.368325 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:54.368341 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:54.368409 (+ 68us) server_negotiation.cc:407] Connection header received
0621 02:42:54.370327 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:54.370338 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:54.370780 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:54.371102 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:54.372564 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:54.373961 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:54.375203 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:54.375861 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:54.376016 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:54.381924 (+ 5908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:54.381932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:54.381948 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:54.381991 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:54.384965 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:54.385790 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:54.385797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:54.385804 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:54.386016 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:54.386370 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:54.386373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:54.386375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:54.386931 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:54.387185 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:54.387620 (+ 435us) server_negotiation.cc:299] Negotiation successful
0621 02:42:54.388071 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":290,"threads_started":1}
I20260621 02:42:54.389859 17157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:54.392339 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:54.391272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46193 (local address 127.0.97.3:40599)
0621 02:42:54.391846 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:54.391984 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
I20260621 02:42:54.477269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 16993
I20260621 02:42:54.495311 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 17165
I20260621 02:42:54.513707 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:54.496274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.0.97.2:40965)
0621 02:42:54.496852 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:54.496866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:54.496903 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:42:54.498507 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:54.498517 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:54.498941 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:54.499237 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:54.500610 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:54.502040 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:54.503352 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:54.504069 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:54.504218 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:54.507096 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:54.507105 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:54.507121 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:54.507164 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:54.510344 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:54.511100 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:54.511106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:54.511113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:54.511284 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:54.511697 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:54.511700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:54.511702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:54.512251 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:54.512476 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:54.512967 (+ 491us) server_negotiation.cc:299] Negotiation successful
0621 02:42:54.513420 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":293,"threads_started":1}
I20260621 02:42:54.515115 17288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:54.599241 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 17165
I20260621 02:42:54.616178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 17296
I20260621 02:42:54.633925 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:54.617160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.0.97.3:46117)
0621 02:42:54.617780 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:54.617794 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:54.617834 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:42:54.619429 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:54.619437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:54.619854 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:54.620223 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:54.621790 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:54.622972 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:54.624223 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:54.624853 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:54.625071 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:54.627758 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:54.627765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:54.627780 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:54.627822 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:54.630758 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:54.631455 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:54.631468 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:54.631473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:54.631623 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:54.632004 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:54.632014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:54.632018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:54.632489 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:54.632797 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:54.633231 (+ 434us) server_negotiation.cc:299] Negotiation successful
0621 02:42:54.633640 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":307,"threads_started":1}
I20260621 02:42:54.635222 17419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:54.720834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 17296
2026-06-21T02:42:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64248 ms)
[----------] 30 tests from SecurityITest (291340 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:42:54 dist-test-slave-jkn7 krb5kdc[17481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:42:54 dist-test-slave-jkn7 krb5kdc[17481](info): set up 2 sockets
Jun 21 02:42:54 dist-test-slave-jkn7 krb5kdc[17481](info): commencing operation
krb5kdc: starting...
W20260621 02:42:55.833520 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:42:55 dist-test-slave-jkn7 krb5kdc[17481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:42:56.066672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42585
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:42:56.387792 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:56.388126 17494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:56.388280 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:56.395377 17494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:42:56.395538 17494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:56.395637 17494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:56.395742 17494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:42:56.395834 17494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:42:56.405372 17494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:56.406654 17494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:56.407960 17494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:56.419813 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:56.420166 17494 server_base.cc:1034] running on GCE node
W20260621 02:42:56.419842 17502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:56.419889 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:56.421193 17494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:56.422550 17494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:56.423782 17494 hybrid_clock.cc:648] HybridClock initialized: now 1782009776423739 us; error 36 us; skew 500 ppm
I20260621 02:42:56.424122 17494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:56 dist-test-slave-jkn7 krb5kdc[17481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:56.427172 17494 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:42:56.428697 17494 webserver.cc:460] Webserver started at http://127.0.97.62:40045/ using document root <none> and password file <none>
I20260621 02:42:56.429405 17494 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:56.429512 17494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:56.429768 17494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:42:56.432703 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3"
format_stamp: "Formatted at 2026-06-21 02:42:56 on dist-test-slave-jkn7"
server_key: "af0417b2463ac8dd1a0e8f21f6dcafce"
server_key_iv: "421e65791b1eab9f20efc58f6559343a"
server_key_version: "encryptionkey@0"
I20260621 02:42:56.433470 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3"
format_stamp: "Formatted at 2026-06-21 02:42:56 on dist-test-slave-jkn7"
server_key: "af0417b2463ac8dd1a0e8f21f6dcafce"
server_key_iv: "421e65791b1eab9f20efc58f6559343a"
server_key_version: "encryptionkey@0"
I20260621 02:42:56.438488 17494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:42:56.441711 17509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:56.442924 17494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:42:56.443100 17494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3"
format_stamp: "Formatted at 2026-06-21 02:42:56 on dist-test-slave-jkn7"
server_key: "af0417b2463ac8dd1a0e8f21f6dcafce"
server_key_iv: "421e65791b1eab9f20efc58f6559343a"
server_key_version: "encryptionkey@0"
I20260621 02:42:56.443284 17494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:56.455466 17494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:56.459117 17494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:56.459430 17494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:56.477965 17494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42585
I20260621 02:42:56.477988 17560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42585 every 8 connection(s)
I20260621 02:42:56.479290 17494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:42:56.482935 17561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:42:56.486692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 17494
I20260621 02:42:56.486901 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:42:56.487727 388 external_mini_cluster.cc:1428] Setting key 852e3d986c10e2f73024a50bdcf685e4
I20260621 02:42:56.496369 17561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3: Bootstrap starting.
I20260621 02:42:56.500180 17561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3: Neither blocks nor log segments found. Creating new log.
I20260621 02:42:56.501555 17561 log.cc:824] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:42:56 dist-test-slave-jkn7 krb5kdc[17481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:42:56.504768 17561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3: No bootstrap required, opened a new log
I20260621 02:42:56.510854 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:56.489791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.0.97.62:42585)
0621 02:42:56.490571 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:56.490583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:56.490622 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:42:56.492189 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:56.492211 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:56.492601 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:56.492905 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:56.494684 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:56.495928 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:56.497786 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:56.498469 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:56.498602 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:56.503724 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:56.503732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:56.503744 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:56.503779 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:56.506853 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:56.507654 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:56.507660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:56.507669 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:56.507849 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:56.508236 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:56.508239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:56.508241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:56.508737 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:56.508959 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:56.509457 (+ 498us) server_negotiation.cc:299] Negotiation successful
0621 02:42:56.509845 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":377,"threads_started":1}
I20260621 02:42:56.514182 17561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260621 02:42:56.514549 17561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:42:56.514658 17561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd813bf9d0e4ecfba3fbc4dfc6a34d3, State: Initialized, Role: FOLLOWER
I20260621 02:42:56.515190 17561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260621 02:42:56.515488 17561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:42:56.515662 17561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:42:56.515861 17561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:42:56.518129 17561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260621 02:42:56.518572 17561 leader_election.cc:302] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd813bf9d0e4ecfba3fbc4dfc6a34d3; no voters:
I20260621 02:42:56.519078 17561 leader_election.cc:288] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:42:56.519241 17566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:42:56.520025 17566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3bd813bf9d0e4ecfba3fbc4dfc6a34d3, State: Running, Role: LEADER
I20260621 02:42:56.520537 17566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260621 02:42:56.521078 17561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:42:56.523948 17568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd813bf9d0e4ecfba3fbc4dfc6a34d3. Latest consensus state: current_term: 1 leader_uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } } }
I20260621 02:42:56.523919 17567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd813bf9d0e4ecfba3fbc4dfc6a34d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } } }
I20260621 02:42:56.524385 17568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:42:56.524385 17567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3 [sys.catalog]: This master's current role is: LEADER
I20260621 02:42:56.531878 17572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:42:56.537163 17572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:42:56.545360 17572 catalog_manager.cc:1347] Generated new cluster ID: 0fc6bff5bce44512bf638758a3ef575c
I20260621 02:42:56.545516 17572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:42:56.554363 17572 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:42:56.556150 17572 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:42:56.570945 17572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3: Generated new TSK 0
I20260621 02:42:56.572001 17572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:42:56.635408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:42:56.967365 17587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:56.967674 17587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:56.967845 17587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:56.975103 17587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:56.975258 17587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:56.975431 17587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:42:56.985064 17587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:56.986343 17587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:56.987627 17587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:56.999825 17592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:56.999881 17593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:56.999917 17595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:57.001267 17587 server_base.cc:1034] running on GCE node
I20260621 02:42:57.001845 17587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:57.002724 17587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:57.003922 17587 hybrid_clock.cc:648] HybridClock initialized: now 1782009777003876 us; error 41 us; skew 500 ppm
I20260621 02:42:57.004238 17587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:57 dist-test-slave-jkn7 krb5kdc[17481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:57.007663 17587 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:42:57.009110 17587 webserver.cc:460] Webserver started at http://127.0.97.1:46461/ using document root <none> and password file <none>
I20260621 02:42:57.009856 17587 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:57.010002 17587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:57.010308 17587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:42:57.013164 17587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "be60ff60b6304104b2c920d139436c86"
format_stamp: "Formatted at 2026-06-21 02:42:57 on dist-test-slave-jkn7"
server_key: "94120e5f24942df8e8ed246e57b0d608"
server_key_iv: "c19b5aa4903a86d1592ecab877aba8e4"
server_key_version: "encryptionkey@0"
I20260621 02:42:57.014042 17587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "be60ff60b6304104b2c920d139436c86"
format_stamp: "Formatted at 2026-06-21 02:42:57 on dist-test-slave-jkn7"
server_key: "94120e5f24942df8e8ed246e57b0d608"
server_key_iv: "c19b5aa4903a86d1592ecab877aba8e4"
server_key_version: "encryptionkey@0"
I20260621 02:42:57.018931 17587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:42:57.022356 17602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:57.023715 17587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:42:57.023921 17587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "be60ff60b6304104b2c920d139436c86"
format_stamp: "Formatted at 2026-06-21 02:42:57 on dist-test-slave-jkn7"
server_key: "94120e5f24942df8e8ed246e57b0d608"
server_key_iv: "c19b5aa4903a86d1592ecab877aba8e4"
server_key_version: "encryptionkey@0"
I20260621 02:42:57.024127 17587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:57.040078 17587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:57.043910 17587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:57.044195 17587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:57.045133 17587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:57.046542 17587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:42:57.046653 17587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:57.046813 17587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:57.046892 17587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:57.081004 17587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41757
I20260621 02:42:57.081092 17714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41757 every 8 connection(s)
I20260621 02:42:57.082296 17587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:42:57.090734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 17587
I20260621 02:42:57.090951 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:42:57.091804 388 external_mini_cluster.cc:1428] Setting key be3824750ebe07d2c2c70e447d9afc22
Jun 21 02:42:57 dist-test-slave-jkn7 krb5kdc[17481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:42:57.102533 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:57.084900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44001 (local address 127.0.97.62:42585)
0621 02:42:57.085400 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:57.085405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:57.086172 (+ 767us) server_negotiation.cc:407] Connection header received
0621 02:42:57.088036 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:57.088042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:57.088131 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:57.088248 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:57.089781 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:57.090515 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:57.092784 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:57.093376 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:57.093508 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:57.096867 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:57.096872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:57.096875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:57.096906 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:57.099497 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:57.100218 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:57.100225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:57.100234 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:57.100354 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:57.100775 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:57.100781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:57.100785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:57.101047 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:57.101203 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:57.101939 (+ 736us) server_negotiation.cc:299] Negotiation successful
0621 02:42:57.102166 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":255,"threads_started":1}
I20260621 02:42:57.103470 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:57.085260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:44001)
0621 02:42:57.086012 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:57.086056 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:57.087754 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:57.088483 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:57.088491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:57.089007 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:42:57.089613 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:57.089629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:57.090675 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:57.090679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:57.092615 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:57.092627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:57.093570 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:57.093575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:57.093861 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:57.094530 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:42:57.094552 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:42:57.096639 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:42:57.099636 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:57.099645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:57.099662 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:57.100091 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:57.100467 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:57.100472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:57.100476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:57.100663 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:57.101178 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:42:57.101185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:42:57.101581 (+ 396us) client_negotiation.cc:769] Sending connection context
0621 02:42:57.101846 (+ 265us) client_negotiation.cc:240] Negotiation successful
0621 02:42:57.102368 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":280,"threads_started":1}
I20260621 02:42:57.105043 17715 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42585
I20260621 02:42:57.105374 17715 heartbeater.cc:464] Registering TS with master...
I20260621 02:42:57.106223 17715 heartbeater.cc:511] Master 127.0.97.62:42585 requested a full tablet report, sending...
I20260621 02:42:57.108618 17526 ts_manager.cc:194] Registered new tserver with Master: be60ff60b6304104b2c920d139436c86 (127.0.97.1:41757)
I20260621 02:42:57.110839 17526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44001
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:42:57.146116 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:42:57.475380 17721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:57.475648 17721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:57.475749 17721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:57.482820 17721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:57.482923 17721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:57.483094 17721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:42:57.492506 17721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:57.493700 17721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:57.494968 17721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:57.507622 17726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:57.507598 17727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:57.507789 17729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:57.508550 17721 server_base.cc:1034] running on GCE node
I20260621 02:42:57.509122 17721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:57.510120 17721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:57.511343 17721 hybrid_clock.cc:648] HybridClock initialized: now 1782009777511311 us; error 54 us; skew 500 ppm
I20260621 02:42:57.511641 17721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:57 dist-test-slave-jkn7 krb5kdc[17481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:57.514907 17721 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:42:57.516363 17721 webserver.cc:460] Webserver started at http://127.0.97.2:42889/ using document root <none> and password file <none>
I20260621 02:42:57.517014 17721 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:57.517153 17721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:57.517503 17721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:42:57.520325 17721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "40148aa73f5146c58c726580fbc0a6b5"
format_stamp: "Formatted at 2026-06-21 02:42:57 on dist-test-slave-jkn7"
server_key: "4d4a7285e7281cbae5a50b21ab828f74"
server_key_iv: "6fadc2bcb80b1878d8a53baa0ac762c0"
server_key_version: "encryptionkey@0"
I20260621 02:42:57.521106 17721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "40148aa73f5146c58c726580fbc0a6b5"
format_stamp: "Formatted at 2026-06-21 02:42:57 on dist-test-slave-jkn7"
server_key: "4d4a7285e7281cbae5a50b21ab828f74"
server_key_iv: "6fadc2bcb80b1878d8a53baa0ac762c0"
server_key_version: "encryptionkey@0"
I20260621 02:42:57.526216 17721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:42:57.529734 17736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:57.531216 17721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:42:57.531390 17721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "40148aa73f5146c58c726580fbc0a6b5"
format_stamp: "Formatted at 2026-06-21 02:42:57 on dist-test-slave-jkn7"
server_key: "4d4a7285e7281cbae5a50b21ab828f74"
server_key_iv: "6fadc2bcb80b1878d8a53baa0ac762c0"
server_key_version: "encryptionkey@0"
I20260621 02:42:57.531594 17721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:57.544222 17721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:57.548722 17721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:57.549036 17721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:57.550066 17721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:57.551426 17721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:42:57.551530 17721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:57.551679 17721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:57.551762 17721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:57.586083 17721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37727
I20260621 02:42:57.586131 17848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37727 every 8 connection(s)
I20260621 02:42:57.587338 17721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:42:57.589185 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 17721
I20260621 02:42:57.589429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:42:57.589948 388 external_mini_cluster.cc:1428] Setting key 676058afcd023690cf8f210b81a8a55e
Jun 21 02:42:57 dist-test-slave-jkn7 krb5kdc[17481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:42:57.611675 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:57.590065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58881 (local address 127.0.97.62:42585)
0621 02:42:57.590284 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:57.590291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:57.591837 (+ 1546us) server_negotiation.cc:407] Connection header received
0621 02:42:57.594695 (+ 2858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:57.594699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:57.594778 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:57.594867 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:57.596920 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:57.597784 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:57.600307 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:57.600775 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:57.600880 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:57.604854 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:57.604863 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:57.604871 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:57.604918 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:57.608580 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:57.609356 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:57.609364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:57.609371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:57.609531 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:57.609975 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:57.609984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:57.609987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:57.610262 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:57.610412 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:57.611092 (+ 680us) server_negotiation.cc:299] Negotiation successful
0621 02:42:57.611333 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:42:57.612661 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:57.590506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:58881)
0621 02:42:57.591684 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:57.591727 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:57.594395 (+ 2668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:57.595134 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:57.595148 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:57.595919 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:42:57.596706 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:57.596732 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:57.597945 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:57.597953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:57.600131 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:57.600150 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:57.601014 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:57.601023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:57.601403 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:57.602164 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:42:57.602187 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:42:57.604607 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:42:57.608738 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:57.608745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:57.608758 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:57.609195 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:57.609647 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:57.609650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:57.609652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:57.609850 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:57.610399 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:42:57.610407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:42:57.610735 (+ 328us) client_negotiation.cc:769] Sending connection context
0621 02:42:57.611009 (+ 274us) client_negotiation.cc:240] Negotiation successful
0621 02:42:57.611546 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":708,"threads_started":1}
I20260621 02:42:57.614209 17849 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42585
I20260621 02:42:57.614560 17849 heartbeater.cc:464] Registering TS with master...
I20260621 02:42:57.615280 17849 heartbeater.cc:511] Master 127.0.97.62:42585 requested a full tablet report, sending...
I20260621 02:42:57.616902 17526 ts_manager.cc:194] Registered new tserver with Master: 40148aa73f5146c58c726580fbc0a6b5 (127.0.97.2:37727)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260621 02:42:57.617837 17526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:42:57.649086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:42:57.984709 17854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:57.984968 17854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:57.985064 17854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:57.992058 17854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:57.992153 17854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:57.992257 17854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:42:58.001605 17854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:58.002741 17854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:58.003954 17854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:58.016513 17859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:58.016542 17862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:58.016618 17860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:58.017524 17854 server_base.cc:1034] running on GCE node
I20260621 02:42:58.018061 17854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:58.018891 17854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:58.020143 17854 hybrid_clock.cc:648] HybridClock initialized: now 1782009778020096 us; error 54 us; skew 500 ppm
I20260621 02:42:58.020481 17854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:42:58 dist-test-slave-jkn7 krb5kdc[17481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:42:58.023658 17854 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:42:58.025022 17854 webserver.cc:460] Webserver started at http://127.0.97.3:46245/ using document root <none> and password file <none>
I20260621 02:42:58.025789 17854 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:58.025930 17854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:58.026234 17854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:42:58.029145 17854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "84508d2a75d04beeb961209a94902d4d"
format_stamp: "Formatted at 2026-06-21 02:42:58 on dist-test-slave-jkn7"
server_key: "522c3babc04d88622beaa63dd96d4030"
server_key_iv: "802fb2c58354019777ea737d6126b41f"
server_key_version: "encryptionkey@0"
I20260621 02:42:58.029954 17854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "84508d2a75d04beeb961209a94902d4d"
format_stamp: "Formatted at 2026-06-21 02:42:58 on dist-test-slave-jkn7"
server_key: "522c3babc04d88622beaa63dd96d4030"
server_key_iv: "802fb2c58354019777ea737d6126b41f"
server_key_version: "encryptionkey@0"
I20260621 02:42:58.035012 17854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:42:58.038527 17869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:58.039805 17854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:42:58.039932 17854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "84508d2a75d04beeb961209a94902d4d"
format_stamp: "Formatted at 2026-06-21 02:42:58 on dist-test-slave-jkn7"
server_key: "522c3babc04d88622beaa63dd96d4030"
server_key_iv: "802fb2c58354019777ea737d6126b41f"
server_key_version: "encryptionkey@0"
I20260621 02:42:58.040089 17854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:58.055295 17854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:58.059136 17854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:58.059384 17854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:58.060259 17854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:58.061537 17854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:42:58.061609 17854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:58.061692 17854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:58.061744 17854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:58.093336 17854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45709
I20260621 02:42:58.093393 17981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45709 every 8 connection(s)
I20260621 02:42:58.094563 17854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:42:58.103665 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 17854
I20260621 02:42:58.103852 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:42:58.104288 388 external_mini_cluster.cc:1428] Setting key 78061181ea67a24801c08c17f3476a1a
Jun 21 02:42:58 dist-test-slave-jkn7 krb5kdc[17481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:42:58.113870 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.097061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47963 (local address 127.0.97.62:42585)
0621 02:42:58.097248 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.097253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.098368 (+ 1115us) server_negotiation.cc:407] Connection header received
0621 02:42:58.100165 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.100171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.100263 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.100379 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:58.101850 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.102629 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.104238 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.104675 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.104802 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.108470 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:58.108475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:58.108478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:58.108517 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:58.111114 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.111809 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.111813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.111815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.111936 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.112370 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.112374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.112375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.112606 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:58.112711 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.113474 (+ 763us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.113632 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:42:58.114722 17715 heartbeater.cc:503] Master 127.0.97.62:42585 was elected leader, sending a full tablet report...
I20260621 02:42:58.115015 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.097509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:47963)
0621 02:42:58.098209 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:58.098249 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:58.099904 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:58.100596 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:58.100603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:58.101087 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:42:58.101683 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.101699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.102782 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.102790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.104090 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.104101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.104906 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.104914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.105158 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.105915 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:42:58.105936 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:42:58.108284 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:42:58.111269 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:58.111275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:58.111288 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:58.111689 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:58.112052 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:42:58.112055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:42:58.112058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:42:58.112268 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:42:58.112738 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:42:58.112744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:42:58.113036 (+ 292us) client_negotiation.cc:769] Sending connection context
0621 02:42:58.113265 (+ 229us) client_negotiation.cc:240] Negotiation successful
0621 02:42:58.113820 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":298,"threads_started":1}
I20260621 02:42:58.116684 17982 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42585
I20260621 02:42:58.116923 17982 heartbeater.cc:464] Registering TS with master...
I20260621 02:42:58.117712 17982 heartbeater.cc:511] Master 127.0.97.62:42585 requested a full tablet report, sending...
I20260621 02:42:58.119184 17526 ts_manager.cc:194] Registered new tserver with Master: 84508d2a75d04beeb961209a94902d4d (127.0.97.3:45709)
I20260621 02:42:58.120019 17526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47963
I20260621 02:42:58.131659 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:42:58.148526 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.136264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.0.97.62:42585)
0621 02:42:58.136435 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.136439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.136872 (+ 433us) server_negotiation.cc:407] Connection header received
0621 02:42:58.137002 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.137006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.137064 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.137145 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:58.138630 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.139320 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.140658 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.141050 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.141150 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.143584 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:58.143589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:58.143592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:58.143633 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:58.146108 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.146812 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.146816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.146818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.146935 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.147292 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.147295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.147297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.147636 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:58.147819 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.148148 (+ 329us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.148314 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:42:58.152465 17526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:42:58.159183 17526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:42:58.181838 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.170432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37727 (local address 127.0.0.1:36932)
0621 02:42:58.172051 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:58.172098 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:58.172249 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:58.174377 (+ 2128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:58.174386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:58.174420 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:58.174790 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.174802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.176918 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.176923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.179171 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.179183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.180766 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.180773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.180946 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.181001 (+ 55us) client_negotiation.cc:769] Sending connection context
0621 02:42:58.181142 (+ 141us) client_negotiation.cc:240] Negotiation successful
0621 02:42:58.181369 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":100,"thread_start_us":1279,"threads_started":1}
I20260621 02:42:58.182168 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.170622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.0.97.2:37727)
0621 02:42:58.173735 (+ 3113us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.173743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.173776 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:42:58.173877 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.173884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.174130 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.174976 (+ 846us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:58.175228 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.176807 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.179328 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.180568 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.180700 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.180897 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.181194 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.181866 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2957,"thread_start_us":2866,"threads_started":1}
I20260621 02:42:58.182587 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.173172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41757 (local address 127.0.0.1:46760)
0621 02:42:58.174000 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:58.174016 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:58.174129 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:58.175271 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:58.175276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:58.175309 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:58.175761 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.175775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.177795 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.177805 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.179953 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.179962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.181777 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.181785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.181957 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.181991 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:42:58.182079 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 02:42:58.182218 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":1,"thread_start_us":871,"threads_started":1}
I20260621 02:42:58.183593 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.170600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45709 (local address 127.0.0.1:59488)
0621 02:42:58.173372 (+ 2772us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:58.173393 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:58.173508 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:58.175030 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:58.175035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:58.175057 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:58.175520 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.175531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.176853 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.176861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.179692 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.179706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.182875 (+ 3169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.182882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.182996 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.183017 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:42:58.183123 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:42:58.183240 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2617,"mutex_wait_us":111,"thread_start_us":2482,"threads_started":1}
I20260621 02:42:58.183642 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.171257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.0.97.3:45709)
0621 02:42:58.172897 (+ 1640us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.172906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.174598 (+ 1692us) server_negotiation.cc:407] Connection header received
0621 02:42:58.174708 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.174713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.174867 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.175042 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:58.175702 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.176715 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.179855 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.181156 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.181277 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.181631 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.183161 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.183306 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":1282,"threads_started":1}
I20260621 02:42:58.183709 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.173839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.0.97.1:41757)
0621 02:42:58.174682 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.174692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.174719 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:42:58.174825 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.174832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.175053 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.175935 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:58.176165 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.177623 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.180097 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.181229 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.181493 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.181710 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.183216 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.183397 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":497,"threads_started":1}
I20260621 02:42:58.186043 17784 tablet_service.cc:1467] Processing CreateTablet for tablet 858f51756f634844800dc6aa4112f562 (DEFAULT_TABLE table=test-table [id=cffefdccdd0846ed87c0ac9593183766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:42:58.187587 17784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858f51756f634844800dc6aa4112f562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:42:58.187333 17650 tablet_service.cc:1467] Processing CreateTablet for tablet 858f51756f634844800dc6aa4112f562 (DEFAULT_TABLE table=test-table [id=cffefdccdd0846ed87c0ac9593183766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:42:58.187526 17917 tablet_service.cc:1467] Processing CreateTablet for tablet 858f51756f634844800dc6aa4112f562 (DEFAULT_TABLE table=test-table [id=cffefdccdd0846ed87c0ac9593183766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:42:58.188661 17650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858f51756f634844800dc6aa4112f562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:42:58.188683 17917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858f51756f634844800dc6aa4112f562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:42:58.199003 18000 tablet_bootstrap.cc:492] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: Bootstrap starting.
I20260621 02:42:58.201686 18001 tablet_bootstrap.cc:492] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: Bootstrap starting.
I20260621 02:42:58.203012 18002 tablet_bootstrap.cc:492] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: Bootstrap starting.
I20260621 02:42:58.203516 18000 tablet_bootstrap.cc:654] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: Neither blocks nor log segments found. Creating new log.
I20260621 02:42:58.204767 18000 log.cc:824] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:42:58.205436 18001 tablet_bootstrap.cc:654] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: Neither blocks nor log segments found. Creating new log.
I20260621 02:42:58.205916 18002 tablet_bootstrap.cc:654] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: Neither blocks nor log segments found. Creating new log.
I20260621 02:42:58.206487 18001 log.cc:824] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:42:58.206866 18002 log.cc:824] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: Log is configured to *not* fsync() on all Append() calls
I20260621 02:42:58.208088 18000 tablet_bootstrap.cc:492] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: No bootstrap required, opened a new log
I20260621 02:42:58.208742 18000 ts_tablet_manager.cc:1397] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260621 02:42:58.209421 18001 tablet_bootstrap.cc:492] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: No bootstrap required, opened a new log
I20260621 02:42:58.209672 18001 ts_tablet_manager.cc:1397] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 02:42:58.209744 18002 tablet_bootstrap.cc:492] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: No bootstrap required, opened a new log
I20260621 02:42:58.210029 18002 ts_tablet_manager.cc:1397] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260621 02:42:58.216111 18000 raft_consensus.cc:348] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } }
I20260621 02:42:58.216468 18000 raft_consensus.cc:374] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:42:58.216583 18000 raft_consensus.cc:729] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84508d2a75d04beeb961209a94902d4d, State: Initialized, Role: FOLLOWER
I20260621 02:42:58.217155 18000 consensus_queue.cc:260] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } }
I20260621 02:42:58.218266 17982 heartbeater.cc:503] Master 127.0.97.62:42585 was elected leader, sending a full tablet report...
I20260621 02:42:58.220012 18000 ts_tablet_manager.cc:1428] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260621 02:42:58.220068 18001 raft_consensus.cc:348] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } }
I20260621 02:42:58.220249 18002 raft_consensus.cc:348] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } }
I20260621 02:42:58.220499 18001 raft_consensus.cc:374] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:42:58.220590 18001 raft_consensus.cc:729] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40148aa73f5146c58c726580fbc0a6b5, State: Initialized, Role: FOLLOWER
I20260621 02:42:58.220624 18002 raft_consensus.cc:374] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:42:58.220708 18002 raft_consensus.cc:729] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be60ff60b6304104b2c920d139436c86, State: Initialized, Role: FOLLOWER
I20260621 02:42:58.221232 18002 consensus_queue.cc:260] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } }
I20260621 02:42:58.221231 18001 consensus_queue.cc:260] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } }
I20260621 02:42:58.222371 17849 heartbeater.cc:503] Master 127.0.97.62:42585 was elected leader, sending a full tablet report...
I20260621 02:42:58.223022 18001 ts_tablet_manager.cc:1428] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260621 02:42:58.223419 18002 ts_tablet_manager.cc:1428] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260621 02:42:58.296538 18006 raft_consensus.cc:482] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:42:58.296823 18006 raft_consensus.cc:504] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } }
I20260621 02:42:58.298532 18006 leader_election.cc:288] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40148aa73f5146c58c726580fbc0a6b5 (127.0.97.2:37727), be60ff60b6304104b2c920d139436c86 (127.0.97.1:41757)
I20260621 02:42:58.304877 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.299048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36313 (local address 127.0.97.1:41757)
0621 02:42:58.299261 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.299267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.299290 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:42:58.299527 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.299534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.299615 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.299743 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:58.300180 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.301157 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.303025 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.304039 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.304150 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.304223 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.304555 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.304649 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:42:58.304994 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.298841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41757 (local address 127.0.97.3:36313)
0621 02:42:58.299165 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:58.299188 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:58.299337 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:58.299765 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:58.299769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:58.299795 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:58.300022 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.300033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.301324 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.301330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.302866 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.302877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.304191 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.304197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.304392 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.304427 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:42:58.304526 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 02:42:58.304669 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":44}
I20260621 02:42:58.305599 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.299074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56753 (local address 127.0.97.2:37727)
0621 02:42:58.299261 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.299267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.299928 (+ 661us) server_negotiation.cc:407] Connection header received
0621 02:42:58.300083 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.300090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.300186 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.300252 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:42:58.300876 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.301815 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.303711 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.304692 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.304822 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.304915 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.305315 (+ 400us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.305426 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:42:58.305811 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.298910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37727 (local address 127.0.97.3:56753)
0621 02:42:58.299807 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0621 02:42:58.299824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:42:58.299955 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:42:58.300295 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:42:58.300299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:42:58.300327 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:42:58.300714 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.300725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.301960 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.301963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.303546 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:42:58.303559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.304917 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.304924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:42:58.305120 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.305153 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:42:58.305244 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 02:42:58.305415 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":1,"thread_start_us":545,"threads_started":1}
I20260621 02:42:58.306226 17670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858f51756f634844800dc6aa4112f562" candidate_uuid: "84508d2a75d04beeb961209a94902d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be60ff60b6304104b2c920d139436c86" is_pre_election: true
I20260621 02:42:58.306723 17670 raft_consensus.cc:2454] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84508d2a75d04beeb961209a94902d4d in term 0.
I20260621 02:42:58.306725 17804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858f51756f634844800dc6aa4112f562" candidate_uuid: "84508d2a75d04beeb961209a94902d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40148aa73f5146c58c726580fbc0a6b5" is_pre_election: true
I20260621 02:42:58.307268 17804 raft_consensus.cc:2454] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84508d2a75d04beeb961209a94902d4d in term 0.
I20260621 02:42:58.307376 17873 leader_election.cc:302] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84508d2a75d04beeb961209a94902d4d, be60ff60b6304104b2c920d139436c86; no voters:
I20260621 02:42:58.307773 18006 raft_consensus.cc:2781] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:42:58.307910 18006 raft_consensus.cc:482] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:42:58.308034 18006 raft_consensus.cc:3037] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:42:58.310300 18006 raft_consensus.cc:504] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } }
I20260621 02:42:58.310904 18006 leader_election.cc:288] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [CANDIDATE]: Term 1 election: Requested vote from peers 40148aa73f5146c58c726580fbc0a6b5 (127.0.97.2:37727), be60ff60b6304104b2c920d139436c86 (127.0.97.1:41757)
I20260621 02:42:58.311311 17804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858f51756f634844800dc6aa4112f562" candidate_uuid: "84508d2a75d04beeb961209a94902d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40148aa73f5146c58c726580fbc0a6b5"
I20260621 02:42:58.311344 17670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858f51756f634844800dc6aa4112f562" candidate_uuid: "84508d2a75d04beeb961209a94902d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be60ff60b6304104b2c920d139436c86"
I20260621 02:42:58.311494 17804 raft_consensus.cc:3037] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:42:58.311553 17670 raft_consensus.cc:3037] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:42:58.313789 17670 raft_consensus.cc:2454] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84508d2a75d04beeb961209a94902d4d in term 1.
I20260621 02:42:58.313788 17804 raft_consensus.cc:2454] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84508d2a75d04beeb961209a94902d4d in term 1.
I20260621 02:42:58.314172 17873 leader_election.cc:302] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84508d2a75d04beeb961209a94902d4d, be60ff60b6304104b2c920d139436c86; no voters:
I20260621 02:42:58.314429 18006 raft_consensus.cc:2781] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:42:58.314812 18006 raft_consensus.cc:686] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 1 LEADER]: Becoming Leader. State: Replica: 84508d2a75d04beeb961209a94902d4d, State: Running, Role: LEADER
I20260621 02:42:58.315253 18006 consensus_queue.cc:237] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } }
I20260621 02:42:58.319765 17525 catalog_manager.cc:5495] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 84508d2a75d04beeb961209a94902d4d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "84508d2a75d04beeb961209a94902d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84508d2a75d04beeb961209a94902d4d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 } health_report { overall_health: UNKNOWN } } }
W20260621 02:42:58.334721 17716 tablet.cc:2369] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:42:58.339310 17850 tablet.cc:2369] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:42:58.346129 17983 tablet.cc:2369] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:42:58.369031 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.363812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.0.97.3:45709)
0621 02:42:58.364038 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.364043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.364060 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:42:58.364243 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.364247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.364323 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.364441 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:42:58.364793 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.365586 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.367076 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.367455 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.367545 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.367959 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:42:58.368432 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:42:58.368546 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.368655 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.368717 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 02:42:58.376742 17670 raft_consensus.cc:1261] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [term 1 FOLLOWER]: Refusing update from remote peer 84508d2a75d04beeb961209a94902d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:42:58.376829 17804 raft_consensus.cc:1261] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [term 1 FOLLOWER]: Refusing update from remote peer 84508d2a75d04beeb961209a94902d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:42:58.377687 18010 consensus_queue.cc:1035] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "40148aa73f5146c58c726580fbc0a6b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:42:58.378059 18006 consensus_queue.cc:1035] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be60ff60b6304104b2c920d139436c86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:42:58.388947 18013 mvcc.cc:205] Tried to move back new op lower bound from 7299112052222005248 to 7299112051982757888. Current Snapshot: MvccSnapshot[applied={T|T < 7299112052222005248}]
I20260621 02:42:58.393674 18015 mvcc.cc:205] Tried to move back new op lower bound from 7299112052222005248 to 7299112051982757888. Current Snapshot: MvccSnapshot[applied={T|T < 7299112052222005248}]
I20260621 02:42:58.402722 17524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:42:58.403050 17524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:42:58.407006 17524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bd813bf9d0e4ecfba3fbc4dfc6a34d3: Sending DeleteTablet for 3 replicas of tablet 858f51756f634844800dc6aa4112f562
I20260621 02:42:58.408203 17784 tablet_service.cc:1514] Processing DeleteTablet for tablet 858f51756f634844800dc6aa4112f562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:42:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36932
I20260621 02:42:58.408236 17917 tablet_service.cc:1514] Processing DeleteTablet for tablet 858f51756f634844800dc6aa4112f562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:42:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59488
I20260621 02:42:58.408264 17650 tablet_service.cc:1514] Processing DeleteTablet for tablet 858f51756f634844800dc6aa4112f562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:42:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46760
I20260621 02:42:58.412290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 17587
I20260621 02:42:58.421542 18023 tablet_replica.cc:330] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: stopping tablet replica
I20260621 02:42:58.422096 18023 raft_consensus.cc:2229] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:42:58.422725 18023 raft_consensus.cc:2258] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:42:58.423545 18024 tablet_replica.cc:330] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: stopping tablet replica
I20260621 02:42:58.424002 18024 raft_consensus.cc:2229] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:42:58.424870 18024 raft_consensus.cc:2258] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:42:58.425694 18023 ts_tablet_manager.cc:1905] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:42:58.426448 18022 tablet_replica.cc:330] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: stopping tablet replica
I20260621 02:42:58.426900 18022 raft_consensus.cc:2229] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:42:58.426963 18024 ts_tablet_manager.cc:1905] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:42:58.427526 18022 raft_consensus.cc:2258] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:42:58.429606 18022 ts_tablet_manager.cc:1905] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:42:58.432858 18023 ts_tablet_manager.cc:1918] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:42:58.433051 18023 log.cc:1196] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/858f51756f634844800dc6aa4112f562
I20260621 02:42:58.433694 18023 ts_tablet_manager.cc:1939] T 858f51756f634844800dc6aa4112f562 P 40148aa73f5146c58c726580fbc0a6b5: Deleting consensus metadata
I20260621 02:42:58.435585 17512 catalog_manager.cc:4841] TS 40148aa73f5146c58c726580fbc0a6b5 (127.0.97.2:37727): tablet 858f51756f634844800dc6aa4112f562 (table test-table [id=cffefdccdd0846ed87c0ac9593183766]) successfully deleted
I20260621 02:42:58.436427 18024 ts_tablet_manager.cc:1918] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:42:58.436617 18024 log.cc:1196] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/858f51756f634844800dc6aa4112f562
I20260621 02:42:58.437222 18024 ts_tablet_manager.cc:1939] T 858f51756f634844800dc6aa4112f562 P 84508d2a75d04beeb961209a94902d4d: Deleting consensus metadata
I20260621 02:42:58.438002 18022 ts_tablet_manager.cc:1918] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:42:58.438215 18022 log.cc:1196] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/858f51756f634844800dc6aa4112f562
Jun 21 02:42:58 dist-test-slave-jkn7 krb5kdc[17481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:42:58.438920 18022 ts_tablet_manager.cc:1939] T 858f51756f634844800dc6aa4112f562 P be60ff60b6304104b2c920d139436c86: Deleting consensus metadata
I20260621 02:42:58.439086 17510 catalog_manager.cc:4841] TS 84508d2a75d04beeb961209a94902d4d (127.0.97.3:45709): tablet 858f51756f634844800dc6aa4112f562 (table test-table [id=cffefdccdd0846ed87c0ac9593183766]) successfully deleted
I20260621 02:42:58.440479 17513 catalog_manager.cc:4841] TS be60ff60b6304104b2c920d139436c86 (127.0.97.1:41757): tablet 858f51756f634844800dc6aa4112f562 (table test-table [id=cffefdccdd0846ed87c0ac9593183766]) successfully deleted
I20260621 02:42:58.445811 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.413654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.0.97.1:41757)
0621 02:42:58.413891 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.413900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.425762 (+ 11862us) server_negotiation.cc:407] Connection header received
0621 02:42:58.426017 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.426025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.426163 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.426284 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:58.428278 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.429449 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.431437 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.432100 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.432225 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.439672 (+ 7447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:58.439688 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:58.439706 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:58.439753 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:58.442952 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.443779 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.443786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.443788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.443921 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.444330 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.444333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.444335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.444759 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:58.444978 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.445371 (+ 393us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.445526 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 02:42:58.446501 17710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:58.550290 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 17587
I20260621 02:42:58.569134 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 17721
Jun 21 02:42:58 dist-test-slave-jkn7 krb5kdc[17481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:42:58.584160 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.569959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.0.97.2:37727)
0621 02:42:58.570140 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.570147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.570167 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:42:58.570390 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.570396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.570473 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.570581 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:58.571827 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.572538 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.573884 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.574276 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.574404 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.578631 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:58.578644 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:58.578660 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:58.578696 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:58.581502 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.582156 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.582163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.582165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.582283 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.582643 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.582649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.582652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.583014 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:58.583323 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.583610 (+ 287us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.583830 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:42:58.584872 17844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:58.682089 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 17721
I20260621 02:42:58.700273 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 17854
Jun 21 02:42:58 dist-test-slave-jkn7 krb5kdc[17481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:42:58.719164 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:58.701078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.0.97.3:45709)
0621 02:42:58.701327 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:58.701333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:58.701356 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:42:58.701494 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:58.701501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:58.701585 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:58.701706 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:58.703073 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.704294 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.705807 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:58.706514 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:58.706669 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:58.711574 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:58.711592 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:58.711613 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:42:58.711672 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:58.715962 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.716772 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.716784 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.716788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.716975 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:42:58.717432 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:42:58.717443 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:42:58.717446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:42:58.717952 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:58.718296 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:58.718511 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 02:42:58.718755 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 02:42:58.719941 17977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:58.845388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 17854
I20260621 02:42:58.865027 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 17494
I20260621 02:42:58.866351 17556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:42:58.976092 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 17494
2026-06-21T02:42:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4257 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T02:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:42:59Z Disabled control of system clock
I20260621 02:42:59.048357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38621
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:42:59.372133 18033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:59.372421 18033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:59.372503 18033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:59.379599 18033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:42:59.379712 18033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:59.379753 18033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:59.379812 18033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:42:59.379855 18033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:42:59.388756 18033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:59.389963 18033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:59.391661 18033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:59.402858 18039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:59.402890 18038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:59.402951 18041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:59.404071 18033 server_base.cc:1034] running on GCE node
I20260621 02:42:59.404865 18033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:59.406327 18033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:59.407544 18033 hybrid_clock.cc:648] HybridClock initialized: now 1782009779407499 us; error 35 us; skew 500 ppm
I20260621 02:42:59.407842 18033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:42:59.415823 18033 webserver.cc:460] Webserver started at http://127.0.97.62:35451/ using document root <none> and password file <none>
I20260621 02:42:59.416565 18033 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:59.416657 18033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:59.416924 18033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:42:59.419853 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "2d511a54935f44cb9b94fbcb2f924782"
format_stamp: "Formatted at 2026-06-21 02:42:59 on dist-test-slave-jkn7"
server_key: "3a7bcc3be91d8d32ca9f23f795743ce2"
server_key_iv: "24babfd18594fcf9b14f9c7c88f6f5dd"
server_key_version: "encryptionkey@0"
I20260621 02:42:59.420581 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "2d511a54935f44cb9b94fbcb2f924782"
format_stamp: "Formatted at 2026-06-21 02:42:59 on dist-test-slave-jkn7"
server_key: "3a7bcc3be91d8d32ca9f23f795743ce2"
server_key_iv: "24babfd18594fcf9b14f9c7c88f6f5dd"
server_key_version: "encryptionkey@0"
I20260621 02:42:59.425418 18033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260621 02:42:59.428642 18047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:59.429993 18033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:42:59.430188 18033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "2d511a54935f44cb9b94fbcb2f924782"
format_stamp: "Formatted at 2026-06-21 02:42:59 on dist-test-slave-jkn7"
server_key: "3a7bcc3be91d8d32ca9f23f795743ce2"
server_key_iv: "24babfd18594fcf9b14f9c7c88f6f5dd"
server_key_version: "encryptionkey@0"
I20260621 02:42:59.430389 18033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:59.453914 18033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:59.454744 18033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:59.455035 18033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:59.473098 18033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38621
I20260621 02:42:59.473124 18098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38621 every 8 connection(s)
I20260621 02:42:59.474416 18033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:42:59.477614 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 18033
I20260621 02:42:59.477804 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:42:59.478072 18099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:42:59.478323 388 external_mini_cluster.cc:1428] Setting key 1051e611c337a718e0b509ddbf5e16c8
I20260621 02:42:59.492914 18099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782: Bootstrap starting.
I20260621 02:42:59.493194 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:42:59.480149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.0.97.62:38621)
0621 02:42:59.481222 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0621 02:42:59.481240 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 02:42:59.481317 (+ 77us) server_negotiation.cc:407] Connection header received
0621 02:42:59.483003 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:42:59.483030 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:42:59.483620 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:42:59.484040 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:42:59.484568 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:59.486601 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:59.487964 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:42:59.488994 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:42:59.489201 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:42:59.490435 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:42:59.490445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:42:59.490464 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:42:59.491180 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:42:59.491270 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:42:59.491295 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:42:59.491403 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:42:59.491632 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 02:42:59.491934 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":360,"threads_started":1}
I20260621 02:42:59.496184 18099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782: Neither blocks nor log segments found. Creating new log.
I20260621 02:42:59.497139 18099 log.cc:824] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782: Log is configured to *not* fsync() on all Append() calls
I20260621 02:42:59.499953 18099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782: No bootstrap required, opened a new log
I20260621 02:42:59.507367 18099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d511a54935f44cb9b94fbcb2f924782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38621 } }
I20260621 02:42:59.507730 18099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:42:59.507803 18099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d511a54935f44cb9b94fbcb2f924782, State: Initialized, Role: FOLLOWER
I20260621 02:42:59.508283 18099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d511a54935f44cb9b94fbcb2f924782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38621 } }
I20260621 02:42:59.508483 18099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:42:59.508626 18099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:42:59.508767 18099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:42:59.511279 18099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d511a54935f44cb9b94fbcb2f924782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38621 } }
I20260621 02:42:59.511760 18099 leader_election.cc:302] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d511a54935f44cb9b94fbcb2f924782; no voters:
I20260621 02:42:59.512271 18099 leader_election.cc:288] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:42:59.512444 18104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:42:59.513175 18104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [term 1 LEADER]: Becoming Leader. State: Replica: 2d511a54935f44cb9b94fbcb2f924782, State: Running, Role: LEADER
I20260621 02:42:59.513787 18104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d511a54935f44cb9b94fbcb2f924782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38621 } }
I20260621 02:42:59.514391 18099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:42:59.516836 18105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d511a54935f44cb9b94fbcb2f924782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d511a54935f44cb9b94fbcb2f924782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38621 } } }
I20260621 02:42:59.518250 18105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [sys.catalog]: This master's current role is: LEADER
I20260621 02:42:59.518046 18106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d511a54935f44cb9b94fbcb2f924782. Latest consensus state: current_term: 1 leader_uuid: "2d511a54935f44cb9b94fbcb2f924782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d511a54935f44cb9b94fbcb2f924782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38621 } } }
I20260621 02:42:59.521931 18106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782 [sys.catalog]: This master's current role is: LEADER
I20260621 02:42:59.522879 18114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:42:59.527776 18114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:42:59.536491 18114 catalog_manager.cc:1347] Generated new cluster ID: 01e9c13704c74b31bdefe1dd95ba97e6
I20260621 02:42:59.536599 18114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:42:59.566097 18114 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:42:59.567337 18114 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:42:59.574229 18114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782: Generated new TSK 0
I20260621 02:42:59.574998 18114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:42:59.579144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38621
--builtin_ntp_servers=127.0.97.20:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:42:59.909623 18123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:42:59.909936 18123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:42:59.910096 18123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:42:59.917944 18123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:42:59.918076 18123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:42:59.918223 18123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:42:59.927306 18123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:42:59.928534 18123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:42:59.930284 18123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:42:59.943631 18128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:59.943663 18131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:42:59.943699 18129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:42:59.944659 18123 server_base.cc:1034] running on GCE node
I20260621 02:42:59.945189 18123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:42:59.946151 18123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:42:59.947371 18123 hybrid_clock.cc:648] HybridClock initialized: now 1782009779947330 us; error 33 us; skew 500 ppm
I20260621 02:42:59.947708 18123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:42:59.950021 18123 webserver.cc:460] Webserver started at http://127.0.97.1:43317/ using document root <none> and password file <none>
I20260621 02:42:59.950704 18123 fs_manager.cc:359] Metadata directory not provided
I20260621 02:42:59.950822 18123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:42:59.951148 18123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:42:59.954018 18123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea"
format_stamp: "Formatted at 2026-06-21 02:42:59 on dist-test-slave-jkn7"
server_key: "bdfaa4b4157ff45ee5bf1a75c1790363"
server_key_iv: "21042f28561f3f719b2a43451be0cf74"
server_key_version: "encryptionkey@0"
I20260621 02:42:59.954809 18123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea"
format_stamp: "Formatted at 2026-06-21 02:42:59 on dist-test-slave-jkn7"
server_key: "bdfaa4b4157ff45ee5bf1a75c1790363"
server_key_iv: "21042f28561f3f719b2a43451be0cf74"
server_key_version: "encryptionkey@0"
I20260621 02:42:59.959806 18123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260621 02:42:59.963284 18137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:59.964658 18123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:42:59.964848 18123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea"
format_stamp: "Formatted at 2026-06-21 02:42:59 on dist-test-slave-jkn7"
server_key: "bdfaa4b4157ff45ee5bf1a75c1790363"
server_key_iv: "21042f28561f3f719b2a43451be0cf74"
server_key_version: "encryptionkey@0"
I20260621 02:42:59.965076 18123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:42:59.983350 18123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:42:59.984210 18123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:42:59.984503 18123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:42:59.985504 18123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:42:59.986969 18123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:42:59.987092 18123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:42:59.987234 18123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:42:59.987323 18123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:00.020761 18123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35057
I20260621 02:43:00.020814 18249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35057 every 8 connection(s)
I20260621 02:43:00.022099 18123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:43:00.023375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 18123
I20260621 02:43:00.023555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:43:00.024082 388 external_mini_cluster.cc:1428] Setting key 97d08e9e3f55de74cf95305feb532949
I20260621 02:43:00.027052 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38621
--builtin_ntp_servers=127.0.97.20:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:43:00.035248 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.024953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35111 (local address 127.0.97.62:38621)
0621 02:43:00.025444 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:00.025449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:00.026853 (+ 1404us) server_negotiation.cc:407] Connection header received
0621 02:43:00.028680 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:00.028686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:00.028778 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:00.028925 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:00.030024 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.030741 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.032281 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.032698 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.032831 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.034477 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:00.034482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:00.034485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:00.034503 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:00.034541 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:00.034555 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:00.034641 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:00.034861 (+ 220us) server_negotiation.cc:299] Negotiation successful
0621 02:43:00.034939 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":260,"threads_started":1}
I20260621 02:43:00.036540 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.025620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38621 (local address 127.0.97.1:35111)
0621 02:43:00.026690 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:00.026733 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:00.028397 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:00.029149 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:00.029157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:00.029243 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:00.029857 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.029875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.030902 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.030907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.032124 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.032136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.032928 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.032938 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.033232 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.034067 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:43:00.034073 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:00.034128 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:43:00.034244 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:00.034660 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:00.034674 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:00.034684 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:43:00.034849 (+ 165us) client_negotiation.cc:240] Negotiation successful
0621 02:43:00.035241 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":366,"threads_started":1}
I20260621 02:43:00.038090 18250 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38621
I20260621 02:43:00.038378 18250 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:00.039160 18250 heartbeater.cc:511] Master 127.0.97.62:38621 requested a full tablet report, sending...
I20260621 02:43:00.041162 18064 ts_manager.cc:194] Registered new tserver with Master: c43c9e7cefc5438d8eb13f86e9bb38ea (127.0.97.1:35057)
I20260621 02:43:00.042630 18064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35111
W20260621 02:43:00.349032 18254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:00.349421 18254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:00.349596 18254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:00.357079 18254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:00.357263 18254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:00.357442 18254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:43:00.366847 18254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:00.368209 18254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:00.370165 18254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:00.383225 18259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:00.383163 18260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:00.383179 18262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:00.384027 18254 server_base.cc:1034] running on GCE node
I20260621 02:43:00.384660 18254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:00.385713 18254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:00.386942 18254 hybrid_clock.cc:648] HybridClock initialized: now 1782009780386914 us; error 54 us; skew 500 ppm
I20260621 02:43:00.387269 18254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:00.389617 18254 webserver.cc:460] Webserver started at http://127.0.97.2:42927/ using document root <none> and password file <none>
I20260621 02:43:00.390327 18254 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:00.390462 18254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:00.390786 18254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:00.393762 18254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "3b8decb755044f9ea811c3d414902a3d"
format_stamp: "Formatted at 2026-06-21 02:43:00 on dist-test-slave-jkn7"
server_key: "37798b1145b05e8d556ad26270740471"
server_key_iv: "ca1f8863adc34c3202ead254ad8e94ed"
server_key_version: "encryptionkey@0"
I20260621 02:43:00.394572 18254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3b8decb755044f9ea811c3d414902a3d"
format_stamp: "Formatted at 2026-06-21 02:43:00 on dist-test-slave-jkn7"
server_key: "37798b1145b05e8d556ad26270740471"
server_key_iv: "ca1f8863adc34c3202ead254ad8e94ed"
server_key_version: "encryptionkey@0"
I20260621 02:43:00.399536 18254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:43:00.402978 18268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:00.404361 18254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:43:00.404562 18254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "3b8decb755044f9ea811c3d414902a3d"
format_stamp: "Formatted at 2026-06-21 02:43:00 on dist-test-slave-jkn7"
server_key: "37798b1145b05e8d556ad26270740471"
server_key_iv: "ca1f8863adc34c3202ead254ad8e94ed"
server_key_version: "encryptionkey@0"
I20260621 02:43:00.404793 18254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:00.429219 18254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:00.430132 18254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:00.430413 18254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:00.431373 18254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:00.432861 18254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:00.432984 18254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:00.433127 18254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:00.433220 18254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:00.466718 18254 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42287
I20260621 02:43:00.466806 18380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42287 every 8 connection(s)
I20260621 02:43:00.467972 18254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:43:00.472361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 18254
I20260621 02:43:00.472539 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:43:00.473106 388 external_mini_cluster.cc:1428] Setting key 1d53a13b6f9a74a77f40f8485a5e2e5b
I20260621 02:43:00.476305 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38621
--builtin_ntp_servers=127.0.97.20:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:43:00.483049 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.470537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36089 (local address 127.0.97.62:38621)
0621 02:43:00.470760 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:00.470767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:00.472320 (+ 1553us) server_negotiation.cc:407] Connection header received
0621 02:43:00.474636 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:00.474642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:00.474739 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:00.474851 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:00.476087 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.477200 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.479697 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.480349 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.480465 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.482175 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:00.482181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:00.482186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:00.482209 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:00.482248 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:00.482263 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:00.482350 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:00.482617 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 02:43:00.482730 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:43:00.484313 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.471025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38621 (local address 127.0.97.2:36089)
0621 02:43:00.472124 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:00.472184 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:00.474385 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:00.475067 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:00.475075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:00.475172 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:00.475907 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.475930 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.477435 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.477444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.479539 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.479556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.480583 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.480592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.480923 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.481803 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:43:00.481809 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:00.481863 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:43:00.481969 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:00.482368 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:00.482381 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:00.482391 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:43:00.482565 (+ 174us) client_negotiation.cc:240] Negotiation successful
0621 02:43:00.482982 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":379,"threads_started":1}
I20260621 02:43:00.485767 18381 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38621
I20260621 02:43:00.486063 18381 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:00.486737 18381 heartbeater.cc:511] Master 127.0.97.62:38621 requested a full tablet report, sending...
I20260621 02:43:00.488091 18064 ts_manager.cc:194] Registered new tserver with Master: 3b8decb755044f9ea811c3d414902a3d (127.0.97.2:42287)
I20260621 02:43:00.488902 18064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:36089
W20260621 02:43:00.804538 18384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:00.804813 18384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:00.804961 18384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:00.812388 18384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:00.812491 18384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:00.812660 18384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:43:00.822126 18384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:00.823406 18384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:00.825176 18384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:00.837853 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:00.837872 18392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:00.838037 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:00.841709 18384 server_base.cc:1034] running on GCE node
I20260621 02:43:00.842326 18384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:00.843384 18384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:00.844583 18384 hybrid_clock.cc:648] HybridClock initialized: now 1782009780844531 us; error 42 us; skew 500 ppm
I20260621 02:43:00.844902 18384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:00.847165 18384 webserver.cc:460] Webserver started at http://127.0.97.3:43477/ using document root <none> and password file <none>
I20260621 02:43:00.847859 18384 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:00.847971 18384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:00.848315 18384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:00.851229 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "70427565e99f450ca76141e25453d8c3"
format_stamp: "Formatted at 2026-06-21 02:43:00 on dist-test-slave-jkn7"
server_key: "157dc73bc6f08db7950d1d825ba56dd8"
server_key_iv: "32fe07f84ff0484a1c5307a97dca538b"
server_key_version: "encryptionkey@0"
I20260621 02:43:00.852041 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "70427565e99f450ca76141e25453d8c3"
format_stamp: "Formatted at 2026-06-21 02:43:00 on dist-test-slave-jkn7"
server_key: "157dc73bc6f08db7950d1d825ba56dd8"
server_key_iv: "32fe07f84ff0484a1c5307a97dca538b"
server_key_version: "encryptionkey@0"
I20260621 02:43:00.857141 18384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:43:00.860498 18398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:00.861894 18384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:43:00.862079 18384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "70427565e99f450ca76141e25453d8c3"
format_stamp: "Formatted at 2026-06-21 02:43:00 on dist-test-slave-jkn7"
server_key: "157dc73bc6f08db7950d1d825ba56dd8"
server_key_iv: "32fe07f84ff0484a1c5307a97dca538b"
server_key_version: "encryptionkey@0"
I20260621 02:43:00.862314 18384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:00.884639 18384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:00.885564 18384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:00.885843 18384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:00.886821 18384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:00.888295 18384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:00.888417 18384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:00.888556 18384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:00.888644 18384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:00.921598 18384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39777
I20260621 02:43:00.921686 18510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39777 every 8 connection(s)
I20260621 02:43:00.922856 18384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:43:00.924688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 18384
I20260621 02:43:00.924896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:43:00.925493 388 external_mini_cluster.cc:1428] Setting key 3f57ed11ecdaa79dbf2737a8718f47f2
I20260621 02:43:00.935393 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.925434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47471 (local address 127.0.97.62:38621)
0621 02:43:00.925627 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:00.925634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:00.927281 (+ 1647us) server_negotiation.cc:407] Connection header received
0621 02:43:00.929196 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:00.929200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:00.929264 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:00.929419 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:00.930415 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.931153 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.932628 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.933015 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.933111 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.934540 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:00.934544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:00.934548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:00.934567 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:00.934596 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:00.934608 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:00.934743 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:00.934970 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 02:43:00.935073 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:43:00.936246 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.926044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38621 (local address 127.0.97.3:47471)
0621 02:43:00.927095 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:00.927149 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:00.928945 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:00.929631 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:00.929638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:00.929716 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:00.930245 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.930259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.931289 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.931292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.932501 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.932512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.933242 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.933247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.933493 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.934212 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:43:00.934216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:00.934284 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:43:00.934387 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:00.934762 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:00.934780 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:00.934786 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:43:00.934959 (+ 173us) client_negotiation.cc:240] Negotiation successful
0621 02:43:00.935305 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":307,"threads_started":1}
I20260621 02:43:00.937629 18511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38621
I20260621 02:43:00.937913 18511 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:00.938701 18511 heartbeater.cc:511] Master 127.0.97.62:38621 requested a full tablet report, sending...
I20260621 02:43:00.940244 18064 ts_manager.cc:194] Registered new tserver with Master: 70427565e99f450ca76141e25453d8c3 (127.0.97.3:39777)
I20260621 02:43:00.941092 18064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:47471
I20260621 02:43:00.952528 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:43:00.962054 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.956662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.0.97.62:38621)
0621 02:43:00.956843 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:00.956848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:00.957279 (+ 431us) server_negotiation.cc:407] Connection header received
0621 02:43:00.957463 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:00.957467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:00.957540 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:00.957624 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:00.958053 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.958719 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.960035 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.960418 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.960510 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.961556 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:00.961560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:00.961563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:00.961579 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:00.961609 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:00.961620 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:00.961695 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:00.961771 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 02:43:00.961835 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:43:00.965888 18064 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:43:00.972791 18064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:43:00.995481 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.986937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.0.97.3:39777)
0621 02:43:00.988446 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:00.988467 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:00.988502 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:43:00.988588 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:00.988598 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:00.988769 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:00.988949 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:00.989719 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.991199 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.993411 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.994335 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.994503 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.994770 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:00.994974 (+ 204us) server_negotiation.cc:299] Negotiation successful
0621 02:43:00.995132 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":372,"threads_started":1}
I20260621 02:43:00.995529 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.986520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39777 (local address 127.0.0.1:59002)
0621 02:43:00.987864 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:00.987880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:00.987975 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:00.988948 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:00.988968 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:00.989003 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:00.989514 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.989526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.991352 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.991357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.993189 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.993207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.994531 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.994537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.994718 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.994774 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 02:43:00.994925 (+ 151us) client_negotiation.cc:240] Negotiation successful
0621 02:43:00.995109 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"mutex_wait_us":618,"thread_start_us":1017,"threads_started":1}
I20260621 02:43:00.996243 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.984643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.0.97.2:42287)
0621 02:43:00.985835 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:00.985847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:00.987278 (+ 1431us) server_negotiation.cc:407] Connection header received
0621 02:43:00.987767 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:00.987777 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:00.988034 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:00.988194 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:00.989368 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.990844 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.993557 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.994894 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.995020 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.995211 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:00.995748 (+ 537us) server_negotiation.cc:299] Negotiation successful
0621 02:43:00.995919 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":935,"threads_started":1}
I20260621 02:43:00.997040 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.984415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35057 (local address 127.0.0.1:50288)
0621 02:43:00.987557 (+ 3142us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:00.987578 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:00.987727 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:00.988501 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:00.988508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:00.988545 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:00.989055 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.989073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.990591 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.990598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.993040 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.993055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.996140 (+ 3085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.996147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.996338 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.996375 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:43:00.996461 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:43:00.996600 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2351,"mutex_wait_us":41,"spinlock_wait_cycles":6784,"thread_start_us":1915,"threads_started":1}
I20260621 02:43:00.997059 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.984117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42287 (local address 127.0.0.1:42232)
0621 02:43:00.986992 (+ 2875us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:00.987169 (+ 177us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:00.987466 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:00.988501 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:00.988508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:00.988551 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:00.989202 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.989214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.990979 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.990985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.993363 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:00.993377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.995343 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.995351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:00.995545 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.995587 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 02:43:00.996483 (+ 896us) client_negotiation.cc:240] Negotiation successful
0621 02:43:00.996600 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2610,"mutex_wait_us":99,"spinlock_wait_cycles":3456,"thread_start_us":2396,"threads_started":1}
I20260621 02:43:00.997817 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:00.985023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.0.97.1:35057)
0621 02:43:00.986237 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:00.986248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:00.987881 (+ 1633us) server_negotiation.cc:407] Connection header received
0621 02:43:00.988006 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:00.988016 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:00.988234 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:00.988421 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:00.989319 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.990466 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.993246 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:00.994716 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:00.994847 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:00.995083 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:00.997307 (+ 2224us) server_negotiation.cc:299] Negotiation successful
0621 02:43:00.997501 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":989,"threads_started":1}
I20260621 02:43:00.999640 18446 tablet_service.cc:1467] Processing CreateTablet for tablet cf5ea8a587e54c5eab11da9ccf0d0ba3 (DEFAULT_TABLE table=test-table [id=3b1dc4e5553b46218980df962a143200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:01.000665 18316 tablet_service.cc:1467] Processing CreateTablet for tablet cf5ea8a587e54c5eab11da9ccf0d0ba3 (DEFAULT_TABLE table=test-table [id=3b1dc4e5553b46218980df962a143200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:01.001247 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf5ea8a587e54c5eab11da9ccf0d0ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:01.001940 18316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf5ea8a587e54c5eab11da9ccf0d0ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:01.002305 18185 tablet_service.cc:1467] Processing CreateTablet for tablet cf5ea8a587e54c5eab11da9ccf0d0ba3 (DEFAULT_TABLE table=test-table [id=3b1dc4e5553b46218980df962a143200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:01.003269 18185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf5ea8a587e54c5eab11da9ccf0d0ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:01.014969 18529 tablet_bootstrap.cc:492] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: Bootstrap starting.
I20260621 02:43:01.016011 18531 tablet_bootstrap.cc:492] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: Bootstrap starting.
I20260621 02:43:01.019645 18529 tablet_bootstrap.cc:654] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:01.019645 18531 tablet_bootstrap.cc:654] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:01.020993 18529 log.cc:824] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:01.021987 18530 tablet_bootstrap.cc:492] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: Bootstrap starting.
I20260621 02:43:01.023962 18531 log.cc:824] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:01.025429 18529 tablet_bootstrap.cc:492] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: No bootstrap required, opened a new log
I20260621 02:43:01.025746 18529 ts_tablet_manager.cc:1397] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260621 02:43:01.025816 18530 tablet_bootstrap.cc:654] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:01.027029 18530 log.cc:824] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:01.030508 18530 tablet_bootstrap.cc:492] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: No bootstrap required, opened a new log
I20260621 02:43:01.030772 18530 ts_tablet_manager.cc:1397] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260621 02:43:01.030889 18531 tablet_bootstrap.cc:492] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: No bootstrap required, opened a new log
I20260621 02:43:01.031147 18531 ts_tablet_manager.cc:1397] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260621 02:43:01.037184 18529 raft_consensus.cc:348] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } }
I20260621 02:43:01.037770 18529 raft_consensus.cc:374] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:01.037916 18529 raft_consensus.cc:729] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43c9e7cefc5438d8eb13f86e9bb38ea, State: Initialized, Role: FOLLOWER
I20260621 02:43:01.038703 18529 consensus_queue.cc:260] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } }
I20260621 02:43:01.039937 18250 heartbeater.cc:503] Master 127.0.97.62:38621 was elected leader, sending a full tablet report...
I20260621 02:43:01.040580 18529 ts_tablet_manager.cc:1428] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: Time spent starting tablet: real 0.015s user 0.011s sys 0.001s
I20260621 02:43:01.041404 18530 raft_consensus.cc:348] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } }
I20260621 02:43:01.041787 18530 raft_consensus.cc:374] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:01.041906 18530 raft_consensus.cc:729] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8decb755044f9ea811c3d414902a3d, State: Initialized, Role: FOLLOWER
I20260621 02:43:01.042500 18530 consensus_queue.cc:260] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } }
I20260621 02:43:01.044008 18381 heartbeater.cc:503] Master 127.0.97.62:38621 was elected leader, sending a full tablet report...
I20260621 02:43:01.044200 18530 ts_tablet_manager.cc:1428] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260621 02:43:01.044111 18531 raft_consensus.cc:348] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } }
I20260621 02:43:01.044513 18531 raft_consensus.cc:374] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:01.044608 18531 raft_consensus.cc:729] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70427565e99f450ca76141e25453d8c3, State: Initialized, Role: FOLLOWER
I20260621 02:43:01.045151 18531 consensus_queue.cc:260] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } }
I20260621 02:43:01.046175 18511 heartbeater.cc:503] Master 127.0.97.62:38621 was elected leader, sending a full tablet report...
I20260621 02:43:01.046847 18531 ts_tablet_manager.cc:1428] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
W20260621 02:43:01.174451 18512 tablet.cc:2369] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:43:01.220026 18382 tablet.cc:2369] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:43:01.274528 18251 tablet.cc:2369] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:43:01.350611 18536 raft_consensus.cc:482] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:43:01.350946 18536 raft_consensus.cc:504] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } }
I20260621 02:43:01.352692 18536 leader_election.cc:288] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c43c9e7cefc5438d8eb13f86e9bb38ea (127.0.97.1:35057), 70427565e99f450ca76141e25453d8c3 (127.0.97.3:39777)
I20260621 02:43:01.361905 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:01.354676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33475 (local address 127.0.97.1:35057)
0621 02:43:01.354901 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:01.354909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:01.354932 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:43:01.355008 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:01.355013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:01.355109 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:01.355208 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:01.356041 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.357134 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.359542 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.360735 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.360857 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:01.360954 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:01.361512 (+ 558us) server_negotiation.cc:299] Negotiation successful
0621 02:43:01.361644 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:43:01.362005 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:01.353047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35057 (local address 127.0.97.2:33475)
0621 02:43:01.354071 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:01.354094 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:01.354256 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:01.355382 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:01.355389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:01.355425 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:01.355879 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:01.355894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.357364 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.357370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:01.359339 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:01.359354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.361083 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.361092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:01.361274 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:01.361359 (+ 85us) client_negotiation.cc:769] Sending connection context
0621 02:43:01.361472 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 02:43:01.361612 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":20,"thread_start_us":1126,"threads_started":1}
I20260621 02:43:01.362175 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:01.353080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39777 (local address 127.0.97.2:33201)
0621 02:43:01.354494 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:01.354514 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:01.354643 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:01.355112 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:01.355117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:01.355141 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:01.355458 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:01.355467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.356937 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.356944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:01.359260 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:01.359274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.361259 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.361266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:01.361492 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:01.361536 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 02:43:01.361632 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 02:43:01.361752 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":4,"thread_start_us":655,"threads_started":1}
I20260621 02:43:01.362709 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:01.353229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33201 (local address 127.0.97.3:39777)
0621 02:43:01.353482 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:01.353490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:01.354624 (+ 1134us) server_negotiation.cc:407] Connection header received
0621 02:43:01.354882 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:01.354890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:01.354971 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:01.355077 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:01.355632 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.356788 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.359511 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.361050 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.361168 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:01.361264 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:01.362451 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0621 02:43:01.362547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260621 02:43:01.363588 18466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5ea8a587e54c5eab11da9ccf0d0ba3" candidate_uuid: "3b8decb755044f9ea811c3d414902a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70427565e99f450ca76141e25453d8c3" is_pre_election: true
I20260621 02:43:01.364193 18466 raft_consensus.cc:2454] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8decb755044f9ea811c3d414902a3d in term 0.
I20260621 02:43:01.364130 18205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5ea8a587e54c5eab11da9ccf0d0ba3" candidate_uuid: "3b8decb755044f9ea811c3d414902a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" is_pre_election: true
I20260621 02:43:01.364645 18205 raft_consensus.cc:2454] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8decb755044f9ea811c3d414902a3d in term 0.
I20260621 02:43:01.364903 18269 leader_election.cc:302] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8decb755044f9ea811c3d414902a3d, 70427565e99f450ca76141e25453d8c3; no voters:
I20260621 02:43:01.365371 18536 raft_consensus.cc:2781] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:43:01.365530 18536 raft_consensus.cc:482] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:43:01.365656 18536 raft_consensus.cc:3037] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:01.367878 18536 raft_consensus.cc:504] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } }
I20260621 02:43:01.368496 18536 leader_election.cc:288] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [CANDIDATE]: Term 1 election: Requested vote from peers c43c9e7cefc5438d8eb13f86e9bb38ea (127.0.97.1:35057), 70427565e99f450ca76141e25453d8c3 (127.0.97.3:39777)
I20260621 02:43:01.368837 18205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5ea8a587e54c5eab11da9ccf0d0ba3" candidate_uuid: "3b8decb755044f9ea811c3d414902a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea"
I20260621 02:43:01.368955 18466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5ea8a587e54c5eab11da9ccf0d0ba3" candidate_uuid: "3b8decb755044f9ea811c3d414902a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70427565e99f450ca76141e25453d8c3"
I20260621 02:43:01.369046 18205 raft_consensus.cc:3037] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:01.369153 18466 raft_consensus.cc:3037] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:01.372265 18466 raft_consensus.cc:2454] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8decb755044f9ea811c3d414902a3d in term 1.
I20260621 02:43:01.372265 18205 raft_consensus.cc:2454] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8decb755044f9ea811c3d414902a3d in term 1.
I20260621 02:43:01.372720 18271 leader_election.cc:302] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b8decb755044f9ea811c3d414902a3d, 70427565e99f450ca76141e25453d8c3, c43c9e7cefc5438d8eb13f86e9bb38ea; no voters:
I20260621 02:43:01.373013 18536 raft_consensus.cc:2781] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:01.373468 18536 raft_consensus.cc:686] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 1 LEADER]: Becoming Leader. State: Replica: 3b8decb755044f9ea811c3d414902a3d, State: Running, Role: LEADER
I20260621 02:43:01.373900 18536 consensus_queue.cc:237] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } }
I20260621 02:43:01.378167 18064 catalog_manager.cc:5495] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8decb755044f9ea811c3d414902a3d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3b8decb755044f9ea811c3d414902a3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8decb755044f9ea811c3d414902a3d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 } health_report { overall_health: UNKNOWN } } }
I20260621 02:43:01.446581 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:01.441263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.0.97.2:42287)
0621 02:43:01.441473 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:01.441482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:01.441547 (+ 65us) server_negotiation.cc:407] Connection header received
0621 02:43:01.441837 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:01.441842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:01.441931 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:01.442056 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:01.442397 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.443104 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.444614 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.445027 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.445121 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:01.445646 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:01.446124 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:01.446241 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:01.446308 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 02:43:01.446365 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 02:43:01.454491 18205 raft_consensus.cc:1261] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [term 1 FOLLOWER]: Refusing update from remote peer 3b8decb755044f9ea811c3d414902a3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:01.454491 18466 raft_consensus.cc:1261] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [term 1 FOLLOWER]: Refusing update from remote peer 3b8decb755044f9ea811c3d414902a3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:01.455588 18540 consensus_queue.cc:1035] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c43c9e7cefc5438d8eb13f86e9bb38ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:01.455953 18536 consensus_queue.cc:1035] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "70427565e99f450ca76141e25453d8c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:01.486375 18064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:01.486634 18064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:01.490285 18064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d511a54935f44cb9b94fbcb2f924782: Sending DeleteTablet for 3 replicas of tablet cf5ea8a587e54c5eab11da9ccf0d0ba3
I20260621 02:43:01.491346 18316 tablet_service.cc:1514] Processing DeleteTablet for tablet cf5ea8a587e54c5eab11da9ccf0d0ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:01 UTC) from {username='slave'} at 127.0.0.1:42232
I20260621 02:43:01.491385 18185 tablet_service.cc:1514] Processing DeleteTablet for tablet cf5ea8a587e54c5eab11da9ccf0d0ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:01 UTC) from {username='slave'} at 127.0.0.1:50288
I20260621 02:43:01.492035 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet cf5ea8a587e54c5eab11da9ccf0d0ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:01 UTC) from {username='slave'} at 127.0.0.1:59002
I20260621 02:43:01.497334 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 18123
I20260621 02:43:01.498535 18554 tablet_replica.cc:330] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: stopping tablet replica
I20260621 02:43:01.499053 18554 raft_consensus.cc:2229] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:01.499673 18554 raft_consensus.cc:2258] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:01.500597 18553 tablet_replica.cc:330] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: stopping tablet replica
I20260621 02:43:01.501026 18553 raft_consensus.cc:2229] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:01.501502 18553 raft_consensus.cc:2258] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:01.503612 18553 ts_tablet_manager.cc:1905] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:01.508790 18552 tablet_replica.cc:330] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: stopping tablet replica
I20260621 02:43:01.509229 18552 raft_consensus.cc:2229] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:43:01.510030 18552 raft_consensus.cc:2258] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:01.510190 18554 ts_tablet_manager.cc:1905] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:01.511023 18553 ts_tablet_manager.cc:1918] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:01.512398 18552 ts_tablet_manager.cc:1905] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:01.512616 18553 log.cc:1196] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/cf5ea8a587e54c5eab11da9ccf0d0ba3
I20260621 02:43:01.513232 18553 ts_tablet_manager.cc:1939] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P c43c9e7cefc5438d8eb13f86e9bb38ea: Deleting consensus metadata
I20260621 02:43:01.516435 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:01.500099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.0.97.1:35057)
0621 02:43:01.500291 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:01.500297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:01.508140 (+ 7843us) server_negotiation.cc:407] Connection header received
0621 02:43:01.508393 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:01.508401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:01.508542 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:01.508649 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:01.509523 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.510638 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.512536 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.513203 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.513327 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:01.514803 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:01.514814 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:01.514830 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:01.515538 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:01.515597 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:01.515622 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:01.515710 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:01.515952 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 02:43:01.516063 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:43:01.516634 18050 catalog_manager.cc:4841] TS c43c9e7cefc5438d8eb13f86e9bb38ea (127.0.97.1:35057): tablet cf5ea8a587e54c5eab11da9ccf0d0ba3 (table test-table [id=3b1dc4e5553b46218980df962a143200]) successfully deleted
I20260621 02:43:01.517154 18554 ts_tablet_manager.cc:1918] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:01.517412 18554 log.cc:1196] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/cf5ea8a587e54c5eab11da9ccf0d0ba3
I20260621 02:43:01.517716 18245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:01.517993 18554 ts_tablet_manager.cc:1939] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 70427565e99f450ca76141e25453d8c3: Deleting consensus metadata
I20260621 02:43:01.519747 18048 catalog_manager.cc:4841] TS 70427565e99f450ca76141e25453d8c3 (127.0.97.3:39777): tablet cf5ea8a587e54c5eab11da9ccf0d0ba3 (table test-table [id=3b1dc4e5553b46218980df962a143200]) successfully deleted
I20260621 02:43:01.524014 18552 ts_tablet_manager.cc:1918] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:01.524214 18552 log.cc:1196] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/cf5ea8a587e54c5eab11da9ccf0d0ba3
I20260621 02:43:01.524763 18552 ts_tablet_manager.cc:1939] T cf5ea8a587e54c5eab11da9ccf0d0ba3 P 3b8decb755044f9ea811c3d414902a3d: Deleting consensus metadata
I20260621 02:43:01.526443 18051 catalog_manager.cc:4841] TS 3b8decb755044f9ea811c3d414902a3d (127.0.97.2:42287): tablet cf5ea8a587e54c5eab11da9ccf0d0ba3 (table test-table [id=3b1dc4e5553b46218980df962a143200]) successfully deleted
I20260621 02:43:01.610287 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 18123
I20260621 02:43:01.628444 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 18254
I20260621 02:43:01.635883 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:01.629271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.0.97.2:42287)
0621 02:43:01.629572 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:01.629579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:01.629600 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:43:01.629699 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:01.629704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:01.629828 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:01.629946 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:01.630352 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.631140 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.632496 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.632946 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.633093 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:01.634381 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:01.634390 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:01.634424 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:01.635062 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:01.635116 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:01.635138 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:01.635265 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:01.635405 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 02:43:01.635509 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260621 02:43:01.636600 18376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:01.749961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 18254
I20260621 02:43:01.768586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 18384
I20260621 02:43:01.775641 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:01.769398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.0.97.3:39777)
0621 02:43:01.769617 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:01.769623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:01.769657 (+ 34us) server_negotiation.cc:407] Connection header received
0621 02:43:01.769873 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:01.769878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:01.769972 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:01.770106 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:01.770491 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.771194 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.772487 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:01.772892 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:01.773002 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:01.774303 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:01.774311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:01.774325 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:01.774886 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:01.774948 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:01.774969 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:01.775059 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:01.775173 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 02:43:01.775259 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:43:01.776291 18506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:01.863018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 18384
I20260621 02:43:01.879760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 18033
I20260621 02:43:01.880810 18094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:01.987680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 18033
2026-06-21T02:43:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3009 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:02 dist-test-slave-jkn7 krb5kdc[18562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:02 dist-test-slave-jkn7 krb5kdc[18562](info): set up 2 sockets
Jun 21 02:43:02 dist-test-slave-jkn7 krb5kdc[18562](info): commencing operation
krb5kdc: starting...
W20260621 02:43:03.089176 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:03 dist-test-slave-jkn7 krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:03.326546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37613
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:03.669144 18575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:03.669456 18575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:03.669549 18575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:03.676575 18575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:03.676676 18575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:03.676723 18575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:03.676772 18575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:03.676818 18575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:03.686362 18575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:03.687565 18575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:03.688766 18575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:03.700557 18583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:03.700688 18581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:03.701001 18580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:03.701653 18575 server_base.cc:1034] running on GCE node
I20260621 02:43:03.702668 18575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:03.704208 18575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:03.705422 18575 hybrid_clock.cc:648] HybridClock initialized: now 1782009783705348 us; error 47 us; skew 500 ppm
I20260621 02:43:03.705780 18575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:03 dist-test-slave-jkn7 krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:03.708956 18575 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:03.710510 18575 webserver.cc:460] Webserver started at http://127.0.97.62:36641/ using document root <none> and password file <none>
I20260621 02:43:03.711194 18575 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:03.711339 18575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:03.711651 18575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:03.714809 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "c6a44459f1764310ae1857842f71303b"
format_stamp: "Formatted at 2026-06-21 02:43:03 on dist-test-slave-jkn7"
server_key: "f5a63ba21e39e121174613cfd9b097c8"
server_key_iv: "9f7ffca20bace7e5542241391ce2b454"
server_key_version: "encryptionkey@0"
I20260621 02:43:03.715560 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "c6a44459f1764310ae1857842f71303b"
format_stamp: "Formatted at 2026-06-21 02:43:03 on dist-test-slave-jkn7"
server_key: "f5a63ba21e39e121174613cfd9b097c8"
server_key_iv: "9f7ffca20bace7e5542241391ce2b454"
server_key_version: "encryptionkey@0"
I20260621 02:43:03.720664 18575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:43:03.723989 18590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:03.725220 18575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:43:03.725417 18575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "c6a44459f1764310ae1857842f71303b"
format_stamp: "Formatted at 2026-06-21 02:43:03 on dist-test-slave-jkn7"
server_key: "f5a63ba21e39e121174613cfd9b097c8"
server_key_iv: "9f7ffca20bace7e5542241391ce2b454"
server_key_version: "encryptionkey@0"
I20260621 02:43:03.725620 18575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:03.744469 18575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:03.748744 18575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:03.749019 18575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:03.767277 18575 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37613
I20260621 02:43:03.767313 18641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37613 every 8 connection(s)
I20260621 02:43:03.768620 18575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:03.772141 18642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:03.778934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 18575
I20260621 02:43:03.779114 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:03.779670 388 external_mini_cluster.cc:1428] Setting key df8c11883413cb0b3d6c39e5f39abde2
I20260621 02:43:03.783206 18642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b: Bootstrap starting.
I20260621 02:43:03.786713 18642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:03.787720 18642 log.cc:824] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:03.790522 18642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b: No bootstrap required, opened a new log
Jun 21 02:43:03 dist-test-slave-jkn7 krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:03.798864 18642 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a44459f1764310ae1857842f71303b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37613 } }
I20260621 02:43:03.799196 18642 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:03.799369 18642 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a44459f1764310ae1857842f71303b, State: Initialized, Role: FOLLOWER
I20260621 02:43:03.799976 18642 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a44459f1764310ae1857842f71303b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37613 } }
I20260621 02:43:03.800238 18642 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:03.800392 18642 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:03.800583 18642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:03.802872 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:03.781602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.0.97.62:37613)
0621 02:43:03.782363 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:03.782375 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:03.782414 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:43:03.784115 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:03.784136 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:03.784516 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:03.784828 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:03.786666 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:03.788062 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:03.789806 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:03.790515 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:03.790720 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:03.795426 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:03.795434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:03.795450 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:03.795488 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:03.798628 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:03.799403 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:03.799410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:03.799416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:03.799629 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:03.800022 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:03.800026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:03.800030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:03.800723 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:03.801034 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:03.801557 (+ 523us) server_negotiation.cc:299] Negotiation successful
0621 02:43:03.801958 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":364,"threads_started":1}
I20260621 02:43:03.803027 18642 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a44459f1764310ae1857842f71303b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37613 } }
I20260621 02:43:03.803555 18642 leader_election.cc:302] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6a44459f1764310ae1857842f71303b; no voters:
I20260621 02:43:03.804175 18642 leader_election.cc:288] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:03.804296 18647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:03.805753 18647 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [term 1 LEADER]: Becoming Leader. State: Replica: c6a44459f1764310ae1857842f71303b, State: Running, Role: LEADER
I20260621 02:43:03.806445 18642 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:03.806331 18647 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a44459f1764310ae1857842f71303b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37613 } }
I20260621 02:43:03.809149 18649 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6a44459f1764310ae1857842f71303b. Latest consensus state: current_term: 1 leader_uuid: "c6a44459f1764310ae1857842f71303b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a44459f1764310ae1857842f71303b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37613 } } }
I20260621 02:43:03.809540 18649 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:03.809917 18648 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6a44459f1764310ae1857842f71303b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a44459f1764310ae1857842f71303b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37613 } } }
I20260621 02:43:03.810245 18648 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:03.811960 18656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:03.816659 18656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:03.824541 18656 catalog_manager.cc:1347] Generated new cluster ID: a289e7e0603d423894c7765c19f77313
I20260621 02:43:03.824661 18656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:03.853945 18656 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:03.855749 18656 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:03.874393 18656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b: Generated new TSK 0
I20260621 02:43:03.875402 18656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:43:03.934842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37613
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:04.251974 18668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:04.252233 18668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:04.252385 18668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:04.260269 18668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:04.260408 18668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:04.260629 18668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:43:04.272738 18668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:04.274087 18668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:04.275353 18668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:04.287948 18676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:04.288044 18674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:04.288048 18673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:04.289259 18668 server_base.cc:1034] running on GCE node
I20260621 02:43:04.289914 18668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:04.290855 18668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:04.292052 18668 hybrid_clock.cc:648] HybridClock initialized: now 1782009784292005 us; error 42 us; skew 500 ppm
I20260621 02:43:04.292354 18668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:04 dist-test-slave-jkn7 krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:04.295535 18668 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:43:04.296924 18668 webserver.cc:460] Webserver started at http://127.0.97.1:43259/ using document root <none> and password file <none>
I20260621 02:43:04.297631 18668 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:04.297768 18668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:04.298061 18668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:04.300805 18668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "cdbb6566b18349a7aac765dd66039d98"
format_stamp: "Formatted at 2026-06-21 02:43:04 on dist-test-slave-jkn7"
server_key: "e628df6db3660a9908d8bed477b601f0"
server_key_iv: "66bb5b7d106f3b808e01365545900070"
server_key_version: "encryptionkey@0"
I20260621 02:43:04.301607 18668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cdbb6566b18349a7aac765dd66039d98"
format_stamp: "Formatted at 2026-06-21 02:43:04 on dist-test-slave-jkn7"
server_key: "e628df6db3660a9908d8bed477b601f0"
server_key_iv: "66bb5b7d106f3b808e01365545900070"
server_key_version: "encryptionkey@0"
I20260621 02:43:04.306620 18668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:43:04.309973 18683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:04.311347 18668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:43:04.311507 18668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "cdbb6566b18349a7aac765dd66039d98"
format_stamp: "Formatted at 2026-06-21 02:43:04 on dist-test-slave-jkn7"
server_key: "e628df6db3660a9908d8bed477b601f0"
server_key_iv: "66bb5b7d106f3b808e01365545900070"
server_key_version: "encryptionkey@0"
I20260621 02:43:04.311693 18668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:04.324949 18668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:04.328771 18668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:04.329059 18668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:04.330083 18668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:04.331326 18668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:04.331405 18668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:04.331494 18668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:04.331542 18668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:04.364442 18668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43707
I20260621 02:43:04.364486 18795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43707 every 8 connection(s)
I20260621 02:43:04.365760 18668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:43:04.375849 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 18668
I20260621 02:43:04.376041 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:43:04.376618 388 external_mini_cluster.cc:1428] Setting key cc02f547994c20b322f294fe5d9c2bda
Jun 21 02:43:04 dist-test-slave-jkn7 krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:04.386755 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:04.368223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34959 (local address 127.0.97.62:37613)
0621 02:43:04.368651 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:04.368656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:04.369612 (+ 956us) server_negotiation.cc:407] Connection header received
0621 02:43:04.371382 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:04.371387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:04.371486 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:04.371603 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:04.373050 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:04.373936 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:04.375558 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:04.376182 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:04.376312 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:04.380045 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:04.380053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:04.380061 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:04.380107 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:04.383585 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:04.384267 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:04.384275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:04.384282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:04.384445 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:04.384832 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:04.384840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:04.384846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:04.385214 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:04.385404 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:04.386011 (+ 607us) server_negotiation.cc:299] Negotiation successful
0621 02:43:04.386257 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":218,"threads_started":1}
I20260621 02:43:04.387331 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:04.368739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37613 (local address 127.0.97.1:34959)
0621 02:43:04.369468 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:04.369505 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:04.371120 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:04.371836 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:04.371843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:04.372351 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:04.372875 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:04.372891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:04.374090 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:04.374095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:04.375391 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:04.375404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:04.376448 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:04.376460 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:04.376771 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:04.377474 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:04.377498 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:04.379828 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:04.383726 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:04.383732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:04.383743 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:04.384149 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:04.384552 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:04.384555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:04.384557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:04.384725 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:04.385376 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:04.385381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:04.385671 (+ 290us) client_negotiation.cc:769] Sending connection context
0621 02:43:04.385890 (+ 219us) client_negotiation.cc:240] Negotiation successful
0621 02:43:04.386246 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":267,"threads_started":1}
I20260621 02:43:04.388861 18796 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37613
I20260621 02:43:04.389165 18796 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:04.389863 18796 heartbeater.cc:511] Master 127.0.97.62:37613 requested a full tablet report, sending...
I20260621 02:43:04.391855 18607 ts_manager.cc:194] Registered new tserver with Master: cdbb6566b18349a7aac765dd66039d98 (127.0.97.1:43707)
I20260621 02:43:04.393239 18607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34959
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:43:04.431984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37613
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:04.756376 18802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:04.756680 18802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:04.756856 18802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:04.763931 18802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:04.764077 18802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:04.764249 18802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:43:04.773629 18802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:04.774822 18802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:04.776053 18802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:04.788879 18807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:04.788980 18808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:04.789021 18810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:04.789884 18802 server_base.cc:1034] running on GCE node
I20260621 02:43:04.790452 18802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:04.791308 18802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:04.792537 18802 hybrid_clock.cc:648] HybridClock initialized: now 1782009784792493 us; error 54 us; skew 500 ppm
I20260621 02:43:04.792841 18802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:04 dist-test-slave-jkn7 krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:04.799360 18802 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:43:04.800866 18802 webserver.cc:460] Webserver started at http://127.0.97.2:35483/ using document root <none> and password file <none>
I20260621 02:43:04.801602 18802 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:04.801744 18802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:04.802050 18802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:04.804885 18802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2"
format_stamp: "Formatted at 2026-06-21 02:43:04 on dist-test-slave-jkn7"
server_key: "c82c2b376624795f0c64f957b2963e7c"
server_key_iv: "8406881a051cfcab0467b7762fb7189b"
server_key_version: "encryptionkey@0"
I20260621 02:43:04.805681 18802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2"
format_stamp: "Formatted at 2026-06-21 02:43:04 on dist-test-slave-jkn7"
server_key: "c82c2b376624795f0c64f957b2963e7c"
server_key_iv: "8406881a051cfcab0467b7762fb7189b"
server_key_version: "encryptionkey@0"
I20260621 02:43:04.810755 18802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 02:43:04.814105 18817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:04.815366 18802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:43:04.815529 18802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2"
format_stamp: "Formatted at 2026-06-21 02:43:04 on dist-test-slave-jkn7"
server_key: "c82c2b376624795f0c64f957b2963e7c"
server_key_iv: "8406881a051cfcab0467b7762fb7189b"
server_key_version: "encryptionkey@0"
I20260621 02:43:04.815719 18802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:04.833391 18802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:04.837423 18802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:04.837700 18802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:04.838586 18802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:04.839831 18802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:04.839936 18802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:04.840085 18802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:04.840157 18802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:04.873612 18802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33081
I20260621 02:43:04.873699 18929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33081 every 8 connection(s)
I20260621 02:43:04.874990 18802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:43:04.884068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 18802
I20260621 02:43:04.884260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:43:04.884688 388 external_mini_cluster.cc:1428] Setting key e206011d4c0e5375264ed37d98bc1456
Jun 21 02:43:04 dist-test-slave-jkn7 krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:04.894873 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:04.877416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45441 (local address 127.0.97.62:37613)
0621 02:43:04.877590 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:04.877596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:04.878717 (+ 1121us) server_negotiation.cc:407] Connection header received
0621 02:43:04.880958 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:04.880963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:04.881055 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:04.881166 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:04.882595 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:04.883472 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:04.885134 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:04.885732 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:04.885873 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:04.889227 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:04.889233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:04.889238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:04.889345 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:04.891876 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:04.892576 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:04.892584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:04.892590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:04.892731 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:04.893156 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:04.893162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:04.893166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:04.893499 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:04.893655 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:04.894302 (+ 647us) server_negotiation.cc:299] Negotiation successful
0621 02:43:04.894533 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:43:04.895794 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:04.877882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37613 (local address 127.0.97.2:45441)
0621 02:43:04.878562 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:04.878604 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:04.880701 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:04.881424 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:04.881432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:04.881911 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:04.882431 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:04.882447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:04.883624 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:04.883630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:04.885000 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:04.885011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:04.885959 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:04.885967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:04.886212 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:04.886953 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:04.886980 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:04.889016 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:04.892019 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:04.892028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:04.892043 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:04.892457 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:04.892843 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:04.892852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:04.892857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:04.893043 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:04.893626 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:04.893634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:04.893953 (+ 319us) client_negotiation.cc:769] Sending connection context
0621 02:43:04.894206 (+ 253us) client_negotiation.cc:240] Negotiation successful
0621 02:43:04.894708 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":269,"threads_started":1}
I20260621 02:43:04.897135 18930 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37613
I20260621 02:43:04.897475 18930 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:04.898175 18930 heartbeater.cc:511] Master 127.0.97.62:37613 requested a full tablet report, sending...
I20260621 02:43:04.899652 18607 ts_manager.cc:194] Registered new tserver with Master: b6aa9e491d1f4f8ca177fb8ae2d859b2 (127.0.97.2:33081)
I20260621 02:43:04.900920 18607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45441
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:43:04.936559 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37613
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:05.252539 18935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:05.252796 18935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:05.252887 18935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:05.260145 18935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:05.260243 18935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:05.260361 18935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:43:05.269593 18935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:05.270712 18935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:05.271914 18935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:05.284875 18940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:05.284807 18941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:05.284852 18943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:05.285666 18935 server_base.cc:1034] running on GCE node
I20260621 02:43:05.286270 18935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:05.287160 18935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:05.288404 18935 hybrid_clock.cc:648] HybridClock initialized: now 1782009785288348 us; error 44 us; skew 500 ppm
I20260621 02:43:05.288749 18935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:05 dist-test-slave-jkn7 krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:05.291972 18935 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:43:05.293467 18935 webserver.cc:460] Webserver started at http://127.0.97.3:43125/ using document root <none> and password file <none>
I20260621 02:43:05.294142 18935 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:05.294282 18935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:05.294582 18935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:05.297384 18935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "40e4ab5000c0466285034910737d19c9"
format_stamp: "Formatted at 2026-06-21 02:43:05 on dist-test-slave-jkn7"
server_key: "53c321594256e55410c5fcf638950664"
server_key_iv: "b3963627d0fd6a322f4f47f9948b77ab"
server_key_version: "encryptionkey@0"
I20260621 02:43:05.298179 18935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "40e4ab5000c0466285034910737d19c9"
format_stamp: "Formatted at 2026-06-21 02:43:05 on dist-test-slave-jkn7"
server_key: "53c321594256e55410c5fcf638950664"
server_key_iv: "b3963627d0fd6a322f4f47f9948b77ab"
server_key_version: "encryptionkey@0"
I20260621 02:43:05.303004 18935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:43:05.306376 18950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:05.307806 18935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:43:05.307982 18935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "40e4ab5000c0466285034910737d19c9"
format_stamp: "Formatted at 2026-06-21 02:43:05 on dist-test-slave-jkn7"
server_key: "53c321594256e55410c5fcf638950664"
server_key_iv: "b3963627d0fd6a322f4f47f9948b77ab"
server_key_version: "encryptionkey@0"
I20260621 02:43:05.308180 18935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:05.323467 18935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:05.327566 18935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:05.327845 18935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:05.328843 18935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:05.330268 18935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:05.330386 18935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:05.330541 18935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:05.330616 18935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:05.366446 18935 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41661
I20260621 02:43:05.366465 19062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41661 every 8 connection(s)
I20260621 02:43:05.367673 18935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:43:05.377689 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 18935
I20260621 02:43:05.377882 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:43:05.378317 388 external_mini_cluster.cc:1428] Setting key 79e90b73687ccf7e3aefd6dc12bf2c4e
Jun 21 02:43:05 dist-test-slave-jkn7 krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:05.387683 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.370330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44557 (local address 127.0.97.62:37613)
0621 02:43:05.370519 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.370524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.371637 (+ 1113us) server_negotiation.cc:407] Connection header received
0621 02:43:05.373504 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.373510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.373610 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.373724 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:05.375224 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.375992 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.377711 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.378216 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.378348 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.382234 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:05.382241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:05.382246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:05.382288 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:05.384717 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:05.385455 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:05.385460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:05.385464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:05.385598 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:05.386028 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:05.386033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:05.386036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:05.386298 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:05.386457 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.387123 (+ 666us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.387335 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:43:05.388551 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.370762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37613 (local address 127.0.97.3:44557)
0621 02:43:05.371463 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:05.371505 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:05.373182 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:05.373969 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:05.373978 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:05.374498 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:05.375049 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.375066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.376153 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.376165 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.377566 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.377577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.378447 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.378458 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.378699 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.379560 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:05.379583 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:05.382031 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:05.384853 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:05.384869 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:05.384885 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:05.385326 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:05.385708 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:05.385713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:05.385717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:05.385909 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:05.386440 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:05.386448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:05.386762 (+ 314us) client_negotiation.cc:769] Sending connection context
0621 02:43:05.387018 (+ 256us) client_negotiation.cc:240] Negotiation successful
0621 02:43:05.387508 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":273,"threads_started":1}
I20260621 02:43:05.389926 19063 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37613
I20260621 02:43:05.390177 19063 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:05.390811 19063 heartbeater.cc:511] Master 127.0.97.62:37613 requested a full tablet report, sending...
I20260621 02:43:05.392369 18606 ts_manager.cc:194] Registered new tserver with Master: 40e4ab5000c0466285034910737d19c9 (127.0.97.3:41661)
I20260621 02:43:05.393424 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:43:05.393466 18606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44557
I20260621 02:43:05.395969 18796 heartbeater.cc:503] Master 127.0.97.62:37613 was elected leader, sending a full tablet report...
I20260621 02:43:05.410095 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.397879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.0.97.62:37613)
0621 02:43:05.398031 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.398035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.398488 (+ 453us) server_negotiation.cc:407] Connection header received
0621 02:43:05.398611 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.398615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.398667 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.398742 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:05.400023 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.400739 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.402259 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.402672 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.402769 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.405209 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:05.405214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:05.405217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:05.405245 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:05.407678 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:05.408424 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:05.408427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:05.408429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:05.408534 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:05.408896 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:05.408898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:05.408901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:05.409162 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:05.409437 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.409749 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.409873 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:43:05.414072 18606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:43:05.420912 18606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:43:05.458791 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.433876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33081 (local address 127.0.0.1:48704)
0621 02:43:05.450752 (+ 16876us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:05.450769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:05.450894 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:05.451516 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:05.451620 (+ 104us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:05.451656 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:05.452151 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.452169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.454098 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.454103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.456164 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.456176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.457798 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.457804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.457993 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.458060 (+ 67us) client_negotiation.cc:769] Sending connection context
0621 02:43:05.458182 (+ 122us) client_negotiation.cc:240] Negotiation successful
0621 02:43:05.458373 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16742,"mutex_wait_us":3,"thread_start_us":14870,"threads_started":1}
I20260621 02:43:05.458906 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.434469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.0.97.2:33081)
0621 02:43:05.449872 (+ 15403us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.449886 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.450861 (+ 975us) server_negotiation.cc:407] Connection header received
0621 02:43:05.451070 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.451080 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.451278 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.451518 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:05.452373 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.453936 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.456328 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.457340 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.457462 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.457670 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.458397 (+ 727us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.458571 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15198,"thread_start_us":15059,"threads_started":1}
I20260621 02:43:05.460136 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.433093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.0.97.1:43707)
0621 02:43:05.451442 (+ 18349us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.451449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.451475 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:43:05.451564 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.451571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.451799 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.451987 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:05.452785 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.454369 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.457334 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.458752 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.459375 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.459605 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.459672 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.459832 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18178,"thread_start_us":18041,"threads_started":1}
I20260621 02:43:05.460378 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.432819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.0.97.3:41661)
0621 02:43:05.450481 (+ 17662us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.450492 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.450534 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:43:05.450664 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.450675 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.451986 (+ 1311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.452176 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:05.452938 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.454418 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.457334 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.458736 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.458860 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.459187 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.459897 (+ 710us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.460044 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17448,"thread_start_us":17278,"threads_started":1}
I20260621 02:43:05.460484 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.432763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43707 (local address 127.0.0.1:44322)
0621 02:43:05.449076 (+ 16313us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:05.449100 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:05.449252 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:05.451987 (+ 2735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:05.451992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:05.452015 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:05.452564 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.452578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.454524 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.454529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.457118 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.457133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.459129 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.459136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.459342 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.459429 (+ 87us) client_negotiation.cc:769] Sending connection context
0621 02:43:05.459513 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 02:43:05.459633 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16136,"mutex_wait_us":217,"thread_start_us":17723,"threads_started":1}
I20260621 02:43:05.460793 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.432750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41661 (local address 127.0.0.1:38160)
0621 02:43:05.448213 (+ 15463us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:05.448263 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:05.448468 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:05.452255 (+ 3787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:05.452259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:05.452278 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:05.452780 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.452790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.454580 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.454585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.457118 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.457133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.459341 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.459347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.459504 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.459532 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 02:43:05.459628 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 02:43:05.459756 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15152,"mutex_wait_us":150,"thread_start_us":15004,"threads_started":1}
I20260621 02:43:05.463153 18865 tablet_service.cc:1467] Processing CreateTablet for tablet 49f193de130142b3b67f1d0ddab71856 (DEFAULT_TABLE table=test-table [id=aff095a669f8487baf9f1afc690ee611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:05.463551 18998 tablet_service.cc:1467] Processing CreateTablet for tablet 49f193de130142b3b67f1d0ddab71856 (DEFAULT_TABLE table=test-table [id=aff095a669f8487baf9f1afc690ee611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:05.463876 18731 tablet_service.cc:1467] Processing CreateTablet for tablet 49f193de130142b3b67f1d0ddab71856 (DEFAULT_TABLE table=test-table [id=aff095a669f8487baf9f1afc690ee611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:05.464654 18865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f193de130142b3b67f1d0ddab71856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:05.464890 18998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f193de130142b3b67f1d0ddab71856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:05.465093 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f193de130142b3b67f1d0ddab71856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:05.476773 19081 tablet_bootstrap.cc:492] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: Bootstrap starting.
I20260621 02:43:05.479861 19082 tablet_bootstrap.cc:492] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: Bootstrap starting.
I20260621 02:43:05.480620 19081 tablet_bootstrap.cc:654] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:05.481993 19083 tablet_bootstrap.cc:492] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: Bootstrap starting.
I20260621 02:43:05.482434 19081 log.cc:824] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:05.485177 19081 tablet_bootstrap.cc:492] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: No bootstrap required, opened a new log
I20260621 02:43:05.485420 19082 tablet_bootstrap.cc:654] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:05.485422 19083 tablet_bootstrap.cc:654] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:05.485469 19081 ts_tablet_manager.cc:1397] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260621 02:43:05.486656 19082 log.cc:824] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:05.486656 19083 log.cc:824] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:05.489212 19082 tablet_bootstrap.cc:492] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: No bootstrap required, opened a new log
I20260621 02:43:05.489500 19082 ts_tablet_manager.cc:1397] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260621 02:43:05.489814 19083 tablet_bootstrap.cc:492] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: No bootstrap required, opened a new log
I20260621 02:43:05.490069 19083 ts_tablet_manager.cc:1397] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260621 02:43:05.495434 19081 raft_consensus.cc:348] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } }
I20260621 02:43:05.495865 19081 raft_consensus.cc:374] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:05.495963 19081 raft_consensus.cc:729] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e4ab5000c0466285034910737d19c9, State: Initialized, Role: FOLLOWER
I20260621 02:43:05.496572 19081 consensus_queue.cc:260] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } }
I20260621 02:43:05.497699 19063 heartbeater.cc:503] Master 127.0.97.62:37613 was elected leader, sending a full tablet report...
I20260621 02:43:05.497655 19082 raft_consensus.cc:348] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } }
I20260621 02:43:05.498080 19082 raft_consensus.cc:374] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:05.498167 19082 raft_consensus.cc:729] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbb6566b18349a7aac765dd66039d98, State: Initialized, Role: FOLLOWER
I20260621 02:43:05.498347 19081 ts_tablet_manager.cc:1428] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260621 02:43:05.499138 19082 consensus_queue.cc:260] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } }
I20260621 02:43:05.500031 19083 raft_consensus.cc:348] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } }
I20260621 02:43:05.500437 19083 raft_consensus.cc:374] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:05.500530 19083 raft_consensus.cc:729] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6aa9e491d1f4f8ca177fb8ae2d859b2, State: Initialized, Role: FOLLOWER
I20260621 02:43:05.500866 19082 ts_tablet_manager.cc:1428] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260621 02:43:05.501201 19083 consensus_queue.cc:260] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } }
I20260621 02:43:05.504108 18930 heartbeater.cc:503] Master 127.0.97.62:37613 was elected leader, sending a full tablet report...
I20260621 02:43:05.504805 19083 ts_tablet_manager.cc:1428] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260621 02:43:05.618008 18797 tablet.cc:2369] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:43:05.619257 19064 tablet.cc:2369] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:43:05.626940 18931 tablet.cc:2369] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:43:05.681474 19087 raft_consensus.cc:482] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:43:05.681793 19087 raft_consensus.cc:504] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } }
I20260621 02:43:05.683525 19087 leader_election.cc:288] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdbb6566b18349a7aac765dd66039d98 (127.0.97.1:43707), b6aa9e491d1f4f8ca177fb8ae2d859b2 (127.0.97.2:33081)
I20260621 02:43:05.689041 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.684069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35025 (local address 127.0.97.2:33081)
0621 02:43:05.684340 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.684346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.684364 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:43:05.684440 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.684444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.684500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.684612 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:05.685047 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.685824 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.687398 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.688158 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.688305 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.688395 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.688663 (+ 268us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.688784 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260621 02:43:05.689041 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.683830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33081 (local address 127.0.97.3:35025)
0621 02:43:05.684121 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:05.684141 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:05.684273 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:05.684662 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:05.684667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:05.684705 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:05.684928 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.684941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.685982 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.685987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.687245 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.687256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.688316 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.688322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.688508 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.688531 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:43:05.688637 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:43:05.688769 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260621 02:43:05.690094 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.683865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53453 (local address 127.0.97.1:43707)
0621 02:43:05.684070 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.684076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.684806 (+ 730us) server_negotiation.cc:407] Connection header received
0621 02:43:05.684976 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.684980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.685034 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.685133 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:05.685678 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.686356 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.687963 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.688836 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.688967 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.689073 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.689732 (+ 659us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.689854 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:43:05.690131 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.683882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43707 (local address 127.0.97.3:53453)
0621 02:43:05.684696 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:05.684712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:05.684827 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:05.685151 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:05.685155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:05.685178 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:05.685530 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.685539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.686497 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.686502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.687801 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:05.687812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.689384 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.689390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:05.689551 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.689593 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 02:43:05.689694 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 02:43:05.689808 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":432,"threads_started":1}
I20260621 02:43:05.690166 18885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f193de130142b3b67f1d0ddab71856" candidate_uuid: "40e4ab5000c0466285034910737d19c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" is_pre_election: true
I20260621 02:43:05.690760 18885 raft_consensus.cc:2454] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e4ab5000c0466285034910737d19c9 in term 0.
I20260621 02:43:05.691154 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f193de130142b3b67f1d0ddab71856" candidate_uuid: "40e4ab5000c0466285034910737d19c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbb6566b18349a7aac765dd66039d98" is_pre_election: true
I20260621 02:43:05.691473 18953 leader_election.cc:302] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e4ab5000c0466285034910737d19c9, b6aa9e491d1f4f8ca177fb8ae2d859b2; no voters:
I20260621 02:43:05.691736 18751 raft_consensus.cc:2454] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e4ab5000c0466285034910737d19c9 in term 0.
I20260621 02:43:05.691906 19087 raft_consensus.cc:2781] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:43:05.692071 19087 raft_consensus.cc:482] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:43:05.692188 19087 raft_consensus.cc:3037] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:05.694453 19087 raft_consensus.cc:504] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } }
I20260621 02:43:05.695024 19087 leader_election.cc:288] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [CANDIDATE]: Term 1 election: Requested vote from peers cdbb6566b18349a7aac765dd66039d98 (127.0.97.1:43707), b6aa9e491d1f4f8ca177fb8ae2d859b2 (127.0.97.2:33081)
I20260621 02:43:05.695487 18885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f193de130142b3b67f1d0ddab71856" candidate_uuid: "40e4ab5000c0466285034910737d19c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2"
I20260621 02:43:05.695468 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f193de130142b3b67f1d0ddab71856" candidate_uuid: "40e4ab5000c0466285034910737d19c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbb6566b18349a7aac765dd66039d98"
I20260621 02:43:05.695672 18885 raft_consensus.cc:3037] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:05.695715 18751 raft_consensus.cc:3037] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:05.697934 18885 raft_consensus.cc:2454] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e4ab5000c0466285034910737d19c9 in term 1.
I20260621 02:43:05.698062 18751 raft_consensus.cc:2454] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e4ab5000c0466285034910737d19c9 in term 1.
I20260621 02:43:05.698302 18953 leader_election.cc:302] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e4ab5000c0466285034910737d19c9, b6aa9e491d1f4f8ca177fb8ae2d859b2; no voters:
I20260621 02:43:05.698570 19087 raft_consensus.cc:2781] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:05.699056 19087 raft_consensus.cc:686] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 1 LEADER]: Becoming Leader. State: Replica: 40e4ab5000c0466285034910737d19c9, State: Running, Role: LEADER
I20260621 02:43:05.699607 19087 consensus_queue.cc:237] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } }
I20260621 02:43:05.703953 18607 catalog_manager.cc:5495] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40e4ab5000c0466285034910737d19c9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "40e4ab5000c0466285034910737d19c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e4ab5000c0466285034910737d19c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 } health_report { overall_health: UNKNOWN } } }
I20260621 02:43:05.760100 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.754261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.0.97.3:41661)
0621 02:43:05.754496 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.754503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.754652 (+ 149us) server_negotiation.cc:407] Connection header received
0621 02:43:05.754896 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.754901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.754980 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.755139 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:05.755533 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.756234 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.757827 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.758229 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.758385 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.758991 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:05.759501 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:05.759699 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.759819 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.759868 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 02:43:05.768697 18751 raft_consensus.cc:1261] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [term 1 FOLLOWER]: Refusing update from remote peer 40e4ab5000c0466285034910737d19c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:05.768778 18885 raft_consensus.cc:1261] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [term 1 FOLLOWER]: Refusing update from remote peer 40e4ab5000c0466285034910737d19c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:05.769924 19087 consensus_queue.cc:1035] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbb6566b18349a7aac765dd66039d98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:05.770313 19091 consensus_queue.cc:1035] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6aa9e491d1f4f8ca177fb8ae2d859b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:05.785435 19094 mvcc.cc:205] Tried to move back new op lower bound from 7299112082498338816 to 7299112082229493760. Current Snapshot: MvccSnapshot[applied={T|T < 7299112082498338816}]
I20260621 02:43:05.787114 19096 mvcc.cc:205] Tried to move back new op lower bound from 7299112082498338816 to 7299112082229493760. Current Snapshot: MvccSnapshot[applied={T|T < 7299112082498338816}]
I20260621 02:43:05.800496 18607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:05.800801 18607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:05.804476 18607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6a44459f1764310ae1857842f71303b: Sending DeleteTablet for 3 replicas of tablet 49f193de130142b3b67f1d0ddab71856
I20260621 02:43:05.805701 18998 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f193de130142b3b67f1d0ddab71856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38160
I20260621 02:43:05.805696 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f193de130142b3b67f1d0ddab71856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44322
I20260621 02:43:05.806334 18865 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f193de130142b3b67f1d0ddab71856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48704
I20260621 02:43:05.810838 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 18668
I20260621 02:43:05.815189 19104 tablet_replica.cc:330] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: stopping tablet replica
I20260621 02:43:05.815645 19104 raft_consensus.cc:2229] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:43:05.816391 19104 raft_consensus.cc:2258] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:05.817206 19105 tablet_replica.cc:330] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: stopping tablet replica
I20260621 02:43:05.817621 19105 raft_consensus.cc:2229] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:05.818102 19105 raft_consensus.cc:2258] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:05.818885 19103 tablet_replica.cc:330] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: stopping tablet replica
I20260621 02:43:05.819298 19103 raft_consensus.cc:2229] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:05.819779 19103 raft_consensus.cc:2258] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:05.820608 19104 ts_tablet_manager.cc:1905] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:05.821766 19105 ts_tablet_manager.cc:1905] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:05.822901 19103 ts_tablet_manager.cc:1905] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:05.827404 19104 ts_tablet_manager.cc:1918] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:05.827569 19104 log.cc:1196] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/49f193de130142b3b67f1d0ddab71856
I20260621 02:43:05.828158 19104 ts_tablet_manager.cc:1939] T 49f193de130142b3b67f1d0ddab71856 P 40e4ab5000c0466285034910737d19c9: Deleting consensus metadata
I20260621 02:43:05.828737 19105 ts_tablet_manager.cc:1918] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:05.828892 19105 log.cc:1196] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/49f193de130142b3b67f1d0ddab71856
I20260621 02:43:05.829528 19105 ts_tablet_manager.cc:1939] T 49f193de130142b3b67f1d0ddab71856 P b6aa9e491d1f4f8ca177fb8ae2d859b2: Deleting consensus metadata
I20260621 02:43:05.830120 18594 catalog_manager.cc:4841] TS 40e4ab5000c0466285034910737d19c9 (127.0.97.3:41661): tablet 49f193de130142b3b67f1d0ddab71856 (table test-table [id=aff095a669f8487baf9f1afc690ee611]) successfully deleted
I20260621 02:43:05.830644 19103 ts_tablet_manager.cc:1918] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:05.830807 19103 log.cc:1196] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/49f193de130142b3b67f1d0ddab71856
I20260621 02:43:05.831225 18593 catalog_manager.cc:4841] TS b6aa9e491d1f4f8ca177fb8ae2d859b2 (127.0.97.2:33081): tablet 49f193de130142b3b67f1d0ddab71856 (table test-table [id=aff095a669f8487baf9f1afc690ee611]) successfully deleted
I20260621 02:43:05.831393 19103 ts_tablet_manager.cc:1939] T 49f193de130142b3b67f1d0ddab71856 P cdbb6566b18349a7aac765dd66039d98: Deleting consensus metadata
I20260621 02:43:05.832691 18592 catalog_manager.cc:4841] TS cdbb6566b18349a7aac765dd66039d98 (127.0.97.1:43707): tablet 49f193de130142b3b67f1d0ddab71856 (table test-table [id=aff095a669f8487baf9f1afc690ee611]) successfully deleted
Jun 21 02:43:05 dist-test-slave-jkn7 krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:43:05.841943 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.812196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.0.97.1:43707)
0621 02:43:05.812442 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.812451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.819046 (+ 6595us) server_negotiation.cc:407] Connection header received
0621 02:43:05.819366 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.819373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.819467 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.819551 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:05.821491 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.822581 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.824661 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.825348 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.825453 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.835097 (+ 9644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:05.835112 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:05.835131 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:05.835176 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:05.839014 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:05.839758 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:05.839768 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:05.839772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:05.839930 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:05.840305 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:05.840310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:05.840315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:05.840777 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:05.841107 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.841318 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.841558 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:43:05.842756 18791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:05.948556 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 18668
I20260621 02:43:05.966607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 18802
Jun 21 02:43:05 dist-test-slave-jkn7 krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:43:05.982928 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:05.967480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.0.97.2:33081)
0621 02:43:05.967673 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:05.967679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:05.967702 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:43:05.967819 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:05.967824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:05.967925 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:05.968040 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:05.969267 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.970421 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.971799 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:05.972388 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:05.972516 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:05.976966 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:05.976980 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:05.976996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:05.977033 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:05.979931 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:05.980775 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:05.980781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:05.980784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:05.980901 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:05.981453 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:05.981461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:05.981466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:05.981899 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:05.982107 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:05.982471 (+ 364us) server_negotiation.cc:299] Negotiation successful
0621 02:43:05.982625 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:43:05.983566 18925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:06.110952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 18802
I20260621 02:43:06.133199 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 18935
Jun 21 02:43:06 dist-test-slave-jkn7 krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:43:06.149653 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:06.134094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.0.97.3:41661)
0621 02:43:06.134295 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:06.134300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:06.134321 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:43:06.134443 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:06.134448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:06.134531 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:06.134655 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:06.135986 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:06.136837 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:06.138351 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:06.138826 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:06.138995 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:06.143356 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:06.143372 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:06.143392 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:06.143429 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:06.146595 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:06.147341 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:06.147352 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:06.147360 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:06.147537 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:06.147973 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:06.147979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:06.147982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:06.148373 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:06.148714 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:06.148993 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 02:43:06.149240 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 02:43:06.150432 19058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:06.272790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 18935
I20260621 02:43:06.293340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 18575
I20260621 02:43:06.294384 18637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:06.403815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 18575
2026-06-21T02:43:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4417 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T02:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:06Z Disabled control of system clock
I20260621 02:43:06.477708 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43193
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:43:06.808229 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:06.808493 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:06.808624 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:06.815673 19114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:06.815766 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:06.815855 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:06.815955 19114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:06.816036 19114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:06.824828 19114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:06.826037 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:06.827725 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:06.839005 19119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:06.838972 19120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:06.839466 19122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:06.839980 19114 server_base.cc:1034] running on GCE node
I20260621 02:43:06.840636 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:06.841934 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:06.843147 19114 hybrid_clock.cc:648] HybridClock initialized: now 1782009786843092 us; error 41 us; skew 500 ppm
I20260621 02:43:06.843521 19114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:06.845793 19114 webserver.cc:460] Webserver started at http://127.0.97.62:34325/ using document root <none> and password file <none>
I20260621 02:43:06.846451 19114 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:06.846561 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:06.846903 19114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:06.849716 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "5221c50d57f44029b5eb79d001da8b56"
format_stamp: "Formatted at 2026-06-21 02:43:06 on dist-test-slave-jkn7"
server_key: "651f04176072e50d5edbe87e8b16d8db"
server_key_iv: "4ba220b79a8ca3d345c25b03f49e084a"
server_key_version: "encryptionkey@0"
I20260621 02:43:06.850502 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "5221c50d57f44029b5eb79d001da8b56"
format_stamp: "Formatted at 2026-06-21 02:43:06 on dist-test-slave-jkn7"
server_key: "651f04176072e50d5edbe87e8b16d8db"
server_key_iv: "4ba220b79a8ca3d345c25b03f49e084a"
server_key_version: "encryptionkey@0"
I20260621 02:43:06.855371 19114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:43:06.858727 19128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:06.859987 19114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:43:06.860159 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "5221c50d57f44029b5eb79d001da8b56"
format_stamp: "Formatted at 2026-06-21 02:43:06 on dist-test-slave-jkn7"
server_key: "651f04176072e50d5edbe87e8b16d8db"
server_key_iv: "4ba220b79a8ca3d345c25b03f49e084a"
server_key_version: "encryptionkey@0"
I20260621 02:43:06.860347 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:06.875880 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:06.876643 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:06.876941 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:06.894263 19114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43193
I20260621 02:43:06.894281 19179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43193 every 8 connection(s)
I20260621 02:43:06.895586 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:06.899288 19180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:06.899307 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 19114
I20260621 02:43:06.899497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:06.899946 388 external_mini_cluster.cc:1428] Setting key 4f352e3d4a58cf2774f1c254a13cf2f1
I20260621 02:43:06.914294 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:06.901806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.0.97.62:43193)
0621 02:43:06.902918 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:06.902933 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:06.902983 (+ 50us) server_negotiation.cc:407] Connection header received
0621 02:43:06.904570 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:06.904602 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:06.905164 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:06.905611 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:06.906146 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:06.908070 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:06.909434 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:06.910456 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:06.910639 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:06.911714 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:06.911727 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:06.911750 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:06.912448 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:06.912541 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:06.912564 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:06.912672 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:06.912827 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 02:43:06.913120 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":375,"threads_started":1}
I20260621 02:43:06.914430 19180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56: Bootstrap starting.
I20260621 02:43:06.917838 19180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:06.918709 19180 log.cc:824] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:06.921036 19180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56: No bootstrap required, opened a new log
I20260621 02:43:06.928145 19180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5221c50d57f44029b5eb79d001da8b56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43193 } }
I20260621 02:43:06.928467 19180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:06.928540 19180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5221c50d57f44029b5eb79d001da8b56, State: Initialized, Role: FOLLOWER
I20260621 02:43:06.928979 19180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5221c50d57f44029b5eb79d001da8b56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43193 } }
I20260621 02:43:06.929173 19180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:06.929327 19180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:06.929466 19180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:06.931404 19180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5221c50d57f44029b5eb79d001da8b56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43193 } }
I20260621 02:43:06.931849 19180 leader_election.cc:302] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5221c50d57f44029b5eb79d001da8b56; no voters:
I20260621 02:43:06.932385 19180 leader_election.cc:288] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:06.932546 19185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:06.933240 19185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [term 1 LEADER]: Becoming Leader. State: Replica: 5221c50d57f44029b5eb79d001da8b56, State: Running, Role: LEADER
I20260621 02:43:06.933832 19185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5221c50d57f44029b5eb79d001da8b56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43193 } }
I20260621 02:43:06.934422 19180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:06.936228 19186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5221c50d57f44029b5eb79d001da8b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5221c50d57f44029b5eb79d001da8b56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43193 } } }
I20260621 02:43:06.936556 19186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:06.938221 19190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:06.938194 19187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5221c50d57f44029b5eb79d001da8b56. Latest consensus state: current_term: 1 leader_uuid: "5221c50d57f44029b5eb79d001da8b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5221c50d57f44029b5eb79d001da8b56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43193 } } }
I20260621 02:43:06.938575 19187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:06.943379 19190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:06.951483 19190 catalog_manager.cc:1347] Generated new cluster ID: 2fac389e95344840a949e17c4c7f00c8
I20260621 02:43:06.951591 19190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:06.977607 19190 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:06.979447 19190 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:06.991628 19190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56: Generated new TSK 0
I20260621 02:43:06.992623 19190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:06.999545 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43193
--builtin_ntp_servers=127.0.97.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:43:07.319391 19204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:07.319650 19204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:07.319789 19204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:07.326777 19204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:07.326875 19204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:07.327040 19204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:43:07.335927 19204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:07.337136 19204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:07.338843 19204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:07.351943 19212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:07.352021 19209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:07.352042 19210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:07.352988 19204 server_base.cc:1034] running on GCE node
I20260621 02:43:07.353734 19204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:07.354940 19204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:07.356246 19204 hybrid_clock.cc:648] HybridClock initialized: now 1782009787356182 us; error 49 us; skew 500 ppm
I20260621 02:43:07.356608 19204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:07.359642 19204 webserver.cc:460] Webserver started at http://127.0.97.1:46809/ using document root <none> and password file <none>
I20260621 02:43:07.360566 19204 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:07.360683 19204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:07.361083 19204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:07.364503 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "921d4477bff94d1fa5910d295c5d6ed1"
format_stamp: "Formatted at 2026-06-21 02:43:07 on dist-test-slave-jkn7"
server_key: "687330aeedc16a8fcf93645c692ad139"
server_key_iv: "966204148e634358dfcedbc4366fdf64"
server_key_version: "encryptionkey@0"
I20260621 02:43:07.365337 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "921d4477bff94d1fa5910d295c5d6ed1"
format_stamp: "Formatted at 2026-06-21 02:43:07 on dist-test-slave-jkn7"
server_key: "687330aeedc16a8fcf93645c692ad139"
server_key_iv: "966204148e634358dfcedbc4366fdf64"
server_key_version: "encryptionkey@0"
I20260621 02:43:07.370201 19204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:43:07.373472 19218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:07.374718 19204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:43:07.374899 19204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "921d4477bff94d1fa5910d295c5d6ed1"
format_stamp: "Formatted at 2026-06-21 02:43:07 on dist-test-slave-jkn7"
server_key: "687330aeedc16a8fcf93645c692ad139"
server_key_iv: "966204148e634358dfcedbc4366fdf64"
server_key_version: "encryptionkey@0"
I20260621 02:43:07.375113 19204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:07.385883 19204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:07.386633 19204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:07.386901 19204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:07.387822 19204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:07.389176 19204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:07.389328 19204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:07.389457 19204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:07.389546 19204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:07.422006 19204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35895
I20260621 02:43:07.422055 19330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35895 every 8 connection(s)
I20260621 02:43:07.423336 19204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:43:07.431802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 19204
I20260621 02:43:07.431993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:43:07.432432 388 external_mini_cluster.cc:1428] Setting key 42591a84c7eb40a5e5b94e764300fb13
I20260621 02:43:07.435606 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43193
--builtin_ntp_servers=127.0.97.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:43:07.437368 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:07.425935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43985 (local address 127.0.97.62:43193)
0621 02:43:07.426458 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:07.426463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:07.427496 (+ 1033us) server_negotiation.cc:407] Connection header received
0621 02:43:07.429541 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:07.429548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:07.429651 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:07.429787 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:07.430979 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:07.431797 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:07.433966 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:07.434585 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:07.434716 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:07.436332 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:07.436342 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:07.436350 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:07.436374 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:07.436426 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:07.436445 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:07.436544 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:07.436811 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 02:43:07.436925 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":304,"threads_started":1}
I20260621 02:43:07.438220 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:07.426369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43193 (local address 127.0.97.1:43985)
0621 02:43:07.427311 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:07.427350 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:07.429185 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:07.430053 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:07.430062 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:07.430169 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:07.430782 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:07.430802 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:07.431971 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:07.431979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:07.433829 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:07.433845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:07.434850 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:07.434861 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:07.435076 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:07.435997 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:43:07.436000 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:07.436040 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:43:07.436126 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:07.436573 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:07.436583 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:07.436589 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:43:07.436766 (+ 177us) client_negotiation.cc:240] Negotiation successful
0621 02:43:07.437051 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":300,"threads_started":1}
I20260621 02:43:07.439669 19331 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43193
I20260621 02:43:07.439986 19331 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:07.440841 19331 heartbeater.cc:511] Master 127.0.97.62:43193 requested a full tablet report, sending...
I20260621 02:43:07.442942 19145 ts_manager.cc:194] Registered new tserver with Master: 921d4477bff94d1fa5910d295c5d6ed1 (127.0.97.1:35895)
I20260621 02:43:07.444428 19145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:43985
W20260621 02:43:07.762687 19335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:07.762943 19335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:07.763024 19335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:07.770160 19335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:07.770264 19335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:07.770370 19335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:43:07.779201 19335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:07.780428 19335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:07.782116 19335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:07.794658 19341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:07.794597 19340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:07.794653 19343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:07.795639 19335 server_base.cc:1034] running on GCE node
I20260621 02:43:07.796164 19335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:07.797003 19335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:07.798224 19335 hybrid_clock.cc:648] HybridClock initialized: now 1782009787798185 us; error 35 us; skew 500 ppm
I20260621 02:43:07.798539 19335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:07.800791 19335 webserver.cc:460] Webserver started at http://127.0.97.2:46447/ using document root <none> and password file <none>
I20260621 02:43:07.801555 19335 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:07.801689 19335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:07.802007 19335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:07.804752 19335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "7510f48d39e742a1bc9a7636ce96a4c0"
format_stamp: "Formatted at 2026-06-21 02:43:07 on dist-test-slave-jkn7"
server_key: "2e968f7ef989788ca7cddad98aee38a9"
server_key_iv: "b002266647755c134851066233328e51"
server_key_version: "encryptionkey@0"
I20260621 02:43:07.805598 19335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7510f48d39e742a1bc9a7636ce96a4c0"
format_stamp: "Formatted at 2026-06-21 02:43:07 on dist-test-slave-jkn7"
server_key: "2e968f7ef989788ca7cddad98aee38a9"
server_key_iv: "b002266647755c134851066233328e51"
server_key_version: "encryptionkey@0"
I20260621 02:43:07.810608 19335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:43:07.813994 19349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:07.815269 19335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:43:07.815443 19335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "7510f48d39e742a1bc9a7636ce96a4c0"
format_stamp: "Formatted at 2026-06-21 02:43:07 on dist-test-slave-jkn7"
server_key: "2e968f7ef989788ca7cddad98aee38a9"
server_key_iv: "b002266647755c134851066233328e51"
server_key_version: "encryptionkey@0"
I20260621 02:43:07.815656 19335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:07.830188 19335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:07.830945 19335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:07.831213 19335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:07.832103 19335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:07.833545 19335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:07.833667 19335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:07.833797 19335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:07.833884 19335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:07.866565 19335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44769
I20260621 02:43:07.866648 19461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44769 every 8 connection(s)
I20260621 02:43:07.867846 19335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:43:07.870851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 19335
I20260621 02:43:07.871012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:43:07.871503 388 external_mini_cluster.cc:1428] Setting key 04bca554d3a352a68de7f0f3a0c41283
I20260621 02:43:07.874433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43193
--builtin_ntp_servers=127.0.97.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:43:07.882562 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:07.870583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60893 (local address 127.0.97.62:43193)
0621 02:43:07.870829 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:07.870836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:07.872357 (+ 1521us) server_negotiation.cc:407] Connection header received
0621 02:43:07.874885 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:07.874889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:07.874964 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:07.875052 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:07.876446 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:07.877141 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:07.879580 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:07.880056 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:07.880163 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:07.881842 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:07.881847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:07.881850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:07.881868 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:07.881900 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:07.881912 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:07.881989 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:07.882242 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 02:43:07.882320 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260621 02:43:07.883649 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:07.871144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43193 (local address 127.0.97.2:60893)
0621 02:43:07.872178 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:07.872222 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:07.874593 (+ 2371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:07.875319 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:07.875333 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:07.875454 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:07.876258 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:07.876283 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:07.877326 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:07.877334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:07.879406 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:07.879422 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:07.880285 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:07.880293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:07.880607 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:07.881460 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:43:07.881466 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:07.881518 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:43:07.881619 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:07.882014 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:07.882027 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:07.882038 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:43:07.882206 (+ 168us) client_negotiation.cc:240] Negotiation successful
0621 02:43:07.882618 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":325,"threads_started":1}
I20260621 02:43:07.884955 19462 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43193
I20260621 02:43:07.885257 19462 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:07.885999 19462 heartbeater.cc:511] Master 127.0.97.62:43193 requested a full tablet report, sending...
I20260621 02:43:07.887338 19145 ts_manager.cc:194] Registered new tserver with Master: 7510f48d39e742a1bc9a7636ce96a4c0 (127.0.97.2:44769)
I20260621 02:43:07.888123 19145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:60893
W20260621 02:43:08.203770 19465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:08.204031 19465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:08.204182 19465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:08.211404 19465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:08.211509 19465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:08.211678 19465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:43:08.220866 19465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:08.222198 19465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:08.223937 19465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:08.236711 19470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:08.236901 19473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:08.237418 19471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:08.238085 19465 server_base.cc:1034] running on GCE node
I20260621 02:43:08.238751 19465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:08.239686 19465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:08.240960 19465 hybrid_clock.cc:648] HybridClock initialized: now 1782009788240910 us; error 43 us; skew 500 ppm
I20260621 02:43:08.241360 19465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:08.243599 19465 webserver.cc:460] Webserver started at http://127.0.97.3:45629/ using document root <none> and password file <none>
I20260621 02:43:08.244310 19465 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:08.244395 19465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:08.244722 19465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:08.247622 19465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "45bc6b1efa0d4407a2d2c083a5e60457"
format_stamp: "Formatted at 2026-06-21 02:43:08 on dist-test-slave-jkn7"
server_key: "1c111834e9cc7c915ecae6d8c69531b3"
server_key_iv: "e1126450797ee4c5b127e100a36c9788"
server_key_version: "encryptionkey@0"
I20260621 02:43:08.248399 19465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "45bc6b1efa0d4407a2d2c083a5e60457"
format_stamp: "Formatted at 2026-06-21 02:43:08 on dist-test-slave-jkn7"
server_key: "1c111834e9cc7c915ecae6d8c69531b3"
server_key_iv: "e1126450797ee4c5b127e100a36c9788"
server_key_version: "encryptionkey@0"
I20260621 02:43:08.253397 19465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:43:08.256588 19479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:08.257927 19465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:43:08.258100 19465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "45bc6b1efa0d4407a2d2c083a5e60457"
format_stamp: "Formatted at 2026-06-21 02:43:08 on dist-test-slave-jkn7"
server_key: "1c111834e9cc7c915ecae6d8c69531b3"
server_key_iv: "e1126450797ee4c5b127e100a36c9788"
server_key_version: "encryptionkey@0"
I20260621 02:43:08.258307 19465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:08.286966 19465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:08.287789 19465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:08.288069 19465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:08.288983 19465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:08.290429 19465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:08.290516 19465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:08.290688 19465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:08.290782 19465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:08.324250 19465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38527
I20260621 02:43:08.324338 19591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38527 every 8 connection(s)
I20260621 02:43:08.325652 19465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:43:08.330049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 19465
I20260621 02:43:08.330228 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:43:08.330629 388 external_mini_cluster.cc:1428] Setting key 363b321ec3e656bb74e0ccf2ecbf1b99
I20260621 02:43:08.338245 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.328287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59081 (local address 127.0.97.62:43193)
0621 02:43:08.328504 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.328509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.329933 (+ 1424us) server_negotiation.cc:407] Connection header received
0621 02:43:08.331996 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.332002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.332092 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.332203 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:08.333211 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.333999 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.335621 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.336045 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.336181 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.337444 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:08.337449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:08.337452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:08.337468 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:08.337496 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:08.337508 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:08.337603 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.337828 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.337920 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:43:08.339207 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.328735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43193 (local address 127.0.97.3:59081)
0621 02:43:08.329796 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:08.329832 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:08.331741 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:08.332431 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:08.332438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:08.332524 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:08.333042 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.333061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.334150 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.334155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.335456 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.335489 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.336265 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.336272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.336490 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.337111 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:43:08.337114 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:08.337153 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:43:08.337221 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:08.337600 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:08.337612 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:08.337621 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:43:08.337802 (+ 181us) client_negotiation.cc:240] Negotiation successful
0621 02:43:08.338147 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":279,"threads_started":1}
I20260621 02:43:08.340480 19592 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43193
I20260621 02:43:08.340781 19592 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:08.341566 19592 heartbeater.cc:511] Master 127.0.97.62:43193 requested a full tablet report, sending...
I20260621 02:43:08.342846 19145 ts_manager.cc:194] Registered new tserver with Master: 45bc6b1efa0d4407a2d2c083a5e60457 (127.0.97.3:38527)
I20260621 02:43:08.343602 19145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:59081
I20260621 02:43:08.345331 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:43:08.354880 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.349434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.0.97.62:43193)
0621 02:43:08.349644 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.349650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.349919 (+ 269us) server_negotiation.cc:407] Connection header received
0621 02:43:08.350041 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.350044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.350099 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.350175 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:08.350597 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.351254 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.352638 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.353012 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.353102 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.354151 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:08.354155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:08.354159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:08.354173 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:08.354199 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:08.354219 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:08.354325 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.354519 (+ 194us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.354588 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:43:08.358651 19144 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:43:08.365576 19144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:43:08.401829 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.377450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.0.97.1:35895)
0621 02:43:08.391672 (+ 14222us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.391680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.392745 (+ 1065us) server_negotiation.cc:407] Connection header received
0621 02:43:08.392872 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.392883 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.393147 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.394124 (+ 977us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:08.394353 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.395805 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.399173 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.400271 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.400410 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.400658 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.401305 (+ 647us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.401476 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14066,"thread_start_us":13937,"threads_started":1}
I20260621 02:43:08.401850 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.377038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35895 (local address 127.0.0.1:34750)
0621 02:43:08.392498 (+ 15460us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:08.392519 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:08.392625 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:08.393435 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:08.393442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:08.393470 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:08.393929 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.393944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.395948 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.395956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.398897 (+ 2941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.398913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.400780 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.400786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.401053 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.401101 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 02:43:08.401237 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 02:43:08.401450 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15326,"mutex_wait_us":182,"thread_start_us":15797,"threads_started":1}
I20260621 02:43:08.402007 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.376850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38527 (local address 127.0.0.1:49920)
0621 02:43:08.392283 (+ 15433us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:08.392339 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:08.392544 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:08.394759 (+ 2215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:08.394764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:08.394793 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:08.395148 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.395160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.397006 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.397011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.398819 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.398834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.400973 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.400981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.401265 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.401418 (+ 153us) client_negotiation.cc:769] Sending connection context
0621 02:43:08.401503 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 02:43:08.401628 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15144,"mutex_wait_us":196,"thread_start_us":16081,"threads_started":1}
I20260621 02:43:08.402729 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.377097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.0.97.3:38527)
0621 02:43:08.391390 (+ 14293us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.391403 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.394257 (+ 2854us) server_negotiation.cc:407] Connection header received
0621 02:43:08.394353 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.394360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.394575 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.394761 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:08.395346 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.396849 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.399011 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.400408 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.400555 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.400816 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.402201 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.402374 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14075,"thread_start_us":13904,"threads_started":1}
I20260621 02:43:08.403286 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.380173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.0.97.2:44769)
0621 02:43:08.392073 (+ 11900us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.392081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.393339 (+ 1258us) server_negotiation.cc:407] Connection header received
0621 02:43:08.393629 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.393637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.393860 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.394031 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:08.394643 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.395898 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.399136 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.400471 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.400583 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.400776 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.401585 (+ 809us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.401742 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11759,"thread_start_us":11649,"threads_started":1}
I20260621 02:43:08.403985 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.379450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44769 (local address 127.0.0.1:51886)
0621 02:43:08.393195 (+ 13745us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:08.393228 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:08.393435 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:08.394198 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:08.394202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:08.394221 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:08.394484 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.394491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.396040 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.396046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.398905 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.398913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.400960 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.400967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.401265 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.401413 (+ 148us) client_negotiation.cc:769] Sending connection context
0621 02:43:08.403536 (+ 2123us) client_negotiation.cc:240] Negotiation successful
0621 02:43:08.403657 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13630,"mutex_wait_us":303,"thread_start_us":12593,"threads_started":1}
I20260621 02:43:08.406186 19527 tablet_service.cc:1467] Processing CreateTablet for tablet 84d8edc15ea7440c8854cecca42a881f (DEFAULT_TABLE table=test-table [id=72e7a949909348abb928efe54c4dcb45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:08.406184 19266 tablet_service.cc:1467] Processing CreateTablet for tablet 84d8edc15ea7440c8854cecca42a881f (DEFAULT_TABLE table=test-table [id=72e7a949909348abb928efe54c4dcb45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:08.407445 19527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d8edc15ea7440c8854cecca42a881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:08.407552 19266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d8edc15ea7440c8854cecca42a881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:08.407382 19397 tablet_service.cc:1467] Processing CreateTablet for tablet 84d8edc15ea7440c8854cecca42a881f (DEFAULT_TABLE table=test-table [id=72e7a949909348abb928efe54c4dcb45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:08.408615 19397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d8edc15ea7440c8854cecca42a881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:08.418694 19610 tablet_bootstrap.cc:492] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: Bootstrap starting.
I20260621 02:43:08.422005 19610 tablet_bootstrap.cc:654] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:08.423601 19610 log.cc:824] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:08.428802 19612 tablet_bootstrap.cc:492] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: Bootstrap starting.
I20260621 02:43:08.432806 19610 tablet_bootstrap.cc:492] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: No bootstrap required, opened a new log
I20260621 02:43:08.433064 19610 ts_tablet_manager.cc:1397] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260621 02:43:08.431135 19611 tablet_bootstrap.cc:492] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: Bootstrap starting.
I20260621 02:43:08.433930 19612 tablet_bootstrap.cc:654] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:08.435314 19612 log.cc:824] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:08.437351 19611 tablet_bootstrap.cc:654] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:08.438836 19611 log.cc:824] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:08.443320 19610 raft_consensus.cc:348] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } }
I20260621 02:43:08.443682 19610 raft_consensus.cc:374] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:08.443810 19610 raft_consensus.cc:729] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bc6b1efa0d4407a2d2c083a5e60457, State: Initialized, Role: FOLLOWER
I20260621 02:43:08.443902 19612 tablet_bootstrap.cc:492] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: No bootstrap required, opened a new log
I20260621 02:43:08.444317 19612 ts_tablet_manager.cc:1397] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260621 02:43:08.444476 19610 consensus_queue.cc:260] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } }
I20260621 02:43:08.444720 19611 tablet_bootstrap.cc:492] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: No bootstrap required, opened a new log
I20260621 02:43:08.445024 19611 ts_tablet_manager.cc:1397] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260621 02:43:08.445716 19592 heartbeater.cc:503] Master 127.0.97.62:43193 was elected leader, sending a full tablet report...
I20260621 02:43:08.447374 19331 heartbeater.cc:503] Master 127.0.97.62:43193 was elected leader, sending a full tablet report...
I20260621 02:43:08.451592 19610 ts_tablet_manager.cc:1428] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: Time spent starting tablet: real 0.018s user 0.009s sys 0.005s
I20260621 02:43:08.455190 19612 raft_consensus.cc:348] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } }
I20260621 02:43:08.455675 19612 raft_consensus.cc:374] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:08.455792 19612 raft_consensus.cc:729] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7510f48d39e742a1bc9a7636ce96a4c0, State: Initialized, Role: FOLLOWER
I20260621 02:43:08.455681 19611 raft_consensus.cc:348] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } }
I20260621 02:43:08.456084 19611 raft_consensus.cc:374] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:08.456204 19611 raft_consensus.cc:729] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 921d4477bff94d1fa5910d295c5d6ed1, State: Initialized, Role: FOLLOWER
I20260621 02:43:08.456557 19612 consensus_queue.cc:260] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } }
I20260621 02:43:08.456749 19611 consensus_queue.cc:260] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } }
I20260621 02:43:08.458339 19462 heartbeater.cc:503] Master 127.0.97.62:43193 was elected leader, sending a full tablet report...
I20260621 02:43:08.458415 19612 ts_tablet_manager.cc:1428] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260621 02:43:08.458405 19611 ts_tablet_manager.cc:1428] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260621 02:43:08.558027 19617 raft_consensus.cc:482] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:43:08.558359 19617 raft_consensus.cc:504] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } }
I20260621 02:43:08.560113 19617 leader_election.cc:288] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45bc6b1efa0d4407a2d2c083a5e60457 (127.0.97.3:38527), 921d4477bff94d1fa5910d295c5d6ed1 (127.0.97.1:35895)
I20260621 02:43:08.567704 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.560523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54565 (local address 127.0.97.1:35895)
0621 02:43:08.560768 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.560774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.561314 (+ 540us) server_negotiation.cc:407] Connection header received
0621 02:43:08.561515 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.561521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.561614 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.561710 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:08.562537 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.563646 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.565975 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.566736 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.566884 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.567001 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.567323 (+ 322us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.567446 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 02:43:08.567703 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.560537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43381 (local address 127.0.97.3:38527)
0621 02:43:08.560779 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.560786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.561284 (+ 498us) server_negotiation.cc:407] Connection header received
0621 02:43:08.561515 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.561521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.561604 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.561730 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:08.562537 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.563654 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.565982 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.566762 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.566905 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.567014 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.567334 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.567458 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:43:08.567809 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.560369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38527 (local address 127.0.97.2:43381)
0621 02:43:08.561144 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:08.561164 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:08.561359 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:08.561745 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:08.561750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:08.561780 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:08.562372 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.562381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.563793 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.563798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.565768 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.565782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.566980 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.566987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.567153 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.567198 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 02:43:08.567308 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 02:43:08.567430 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":163,"thread_start_us":323,"threads_started":1}
I20260621 02:43:08.567881 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.560369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35895 (local address 127.0.97.2:54565)
0621 02:43:08.561204 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:08.561219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:08.561379 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:08.561745 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:08.561750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:08.561790 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:08.562321 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.562340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.563794 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.563798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.565761 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:08.565774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.566964 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.566971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:08.567148 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.567186 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 02:43:08.567293 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 02:43:08.567425 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":341,"spinlock_wait_cycles":50816,"thread_start_us":360,"threads_started":1}
I20260621 02:43:08.569054 19547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d8edc15ea7440c8854cecca42a881f" candidate_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" is_pre_election: true
I20260621 02:43:08.569144 19286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d8edc15ea7440c8854cecca42a881f" candidate_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921d4477bff94d1fa5910d295c5d6ed1" is_pre_election: true
I20260621 02:43:08.569600 19547 raft_consensus.cc:2454] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7510f48d39e742a1bc9a7636ce96a4c0 in term 0.
I20260621 02:43:08.569691 19286 raft_consensus.cc:2454] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7510f48d39e742a1bc9a7636ce96a4c0 in term 0.
I20260621 02:43:08.570276 19350 leader_election.cc:302] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45bc6b1efa0d4407a2d2c083a5e60457, 7510f48d39e742a1bc9a7636ce96a4c0, 921d4477bff94d1fa5910d295c5d6ed1; no voters:
I20260621 02:43:08.570647 19617 raft_consensus.cc:2781] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:43:08.570803 19617 raft_consensus.cc:482] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:43:08.570906 19617 raft_consensus.cc:3037] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:08.573071 19617 raft_consensus.cc:504] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } }
I20260621 02:43:08.573741 19617 leader_election.cc:288] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 45bc6b1efa0d4407a2d2c083a5e60457 (127.0.97.3:38527), 921d4477bff94d1fa5910d295c5d6ed1 (127.0.97.1:35895)
I20260621 02:43:08.574195 19286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d8edc15ea7440c8854cecca42a881f" candidate_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921d4477bff94d1fa5910d295c5d6ed1"
I20260621 02:43:08.574195 19547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d8edc15ea7440c8854cecca42a881f" candidate_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bc6b1efa0d4407a2d2c083a5e60457"
I20260621 02:43:08.574398 19286 raft_consensus.cc:3037] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:08.574398 19547 raft_consensus.cc:3037] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [term 0 FOLLOWER]: Advancing to term 1
W20260621 02:43:08.577342 19593 tablet.cc:2369] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:43:08.577595 19547 raft_consensus.cc:2454] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7510f48d39e742a1bc9a7636ce96a4c0 in term 1.
I20260621 02:43:08.577589 19286 raft_consensus.cc:2454] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7510f48d39e742a1bc9a7636ce96a4c0 in term 1.
I20260621 02:43:08.577975 19350 leader_election.cc:302] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45bc6b1efa0d4407a2d2c083a5e60457, 7510f48d39e742a1bc9a7636ce96a4c0; no voters:
I20260621 02:43:08.578253 19617 raft_consensus.cc:2781] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:08.578727 19617 raft_consensus.cc:686] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 7510f48d39e742a1bc9a7636ce96a4c0, State: Running, Role: LEADER
I20260621 02:43:08.579314 19617 consensus_queue.cc:237] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } }
I20260621 02:43:08.583565 19145 catalog_manager.cc:5495] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7510f48d39e742a1bc9a7636ce96a4c0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7510f48d39e742a1bc9a7636ce96a4c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44769 } health_report { overall_health: HEALTHY } } }
I20260621 02:43:08.602612 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.597270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.0.97.2:44769)
0621 02:43:08.597488 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.597493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.597514 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:43:08.597594 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.597598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.597694 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.597824 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:08.598153 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.598871 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.600288 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.600722 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.600827 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.601421 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:08.602007 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:08.602151 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.602343 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.602383 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:43:08.610812 19547 raft_consensus.cc:1261] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [term 1 FOLLOWER]: Refusing update from remote peer 7510f48d39e742a1bc9a7636ce96a4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:08.610812 19286 raft_consensus.cc:1261] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [term 1 FOLLOWER]: Refusing update from remote peer 7510f48d39e742a1bc9a7636ce96a4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:08.611770 19621 consensus_queue.cc:1035] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45bc6b1efa0d4407a2d2c083a5e60457" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:08.612099 19617 consensus_queue.cc:1035] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "921d4477bff94d1fa5910d295c5d6ed1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:43:08.619958 19463 tablet.cc:2369] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:43:08.625586 19624 mvcc.cc:205] Tried to move back new op lower bound from 7299112094140014592 to 7299112094024818688. Current Snapshot: MvccSnapshot[applied={T|T < 7299112094140014592}]
I20260621 02:43:08.643106 19144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:08.643409 19144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:08.646905 19144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5221c50d57f44029b5eb79d001da8b56: Sending DeleteTablet for 3 replicas of tablet 84d8edc15ea7440c8854cecca42a881f
I20260621 02:43:08.648054 19266 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d8edc15ea7440c8854cecca42a881f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:08 UTC) from {username='slave'} at 127.0.0.1:34750
I20260621 02:43:08.648069 19527 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d8edc15ea7440c8854cecca42a881f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:08 UTC) from {username='slave'} at 127.0.0.1:49920
I20260621 02:43:08.648767 19397 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d8edc15ea7440c8854cecca42a881f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:08 UTC) from {username='slave'} at 127.0.0.1:51886
I20260621 02:43:08.648906 19635 tablet_replica.cc:330] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: stopping tablet replica
I20260621 02:43:08.649369 19635 raft_consensus.cc:2229] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:08.649984 19635 raft_consensus.cc:2258] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:08.652060 19635 ts_tablet_manager.cc:1905] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:08.652638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 19204
I20260621 02:43:08.659070 19636 tablet_replica.cc:330] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: stopping tablet replica
I20260621 02:43:08.659528 19636 raft_consensus.cc:2229] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:08.659888 19635 ts_tablet_manager.cc:1918] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:08.660128 19635 log.cc:1196] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/84d8edc15ea7440c8854cecca42a881f
I20260621 02:43:08.660244 19636 raft_consensus.cc:2258] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:08.660707 19635 ts_tablet_manager.cc:1939] T 84d8edc15ea7440c8854cecca42a881f P 921d4477bff94d1fa5910d295c5d6ed1: Deleting consensus metadata
I20260621 02:43:08.662564 19636 ts_tablet_manager.cc:1905] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:08.662765 19132 catalog_manager.cc:4841] TS 921d4477bff94d1fa5910d295c5d6ed1 (127.0.97.1:35895): tablet 84d8edc15ea7440c8854cecca42a881f (table test-table [id=72e7a949909348abb928efe54c4dcb45]) successfully deleted
I20260621 02:43:08.669400 19636 ts_tablet_manager.cc:1918] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:08.669625 19636 log.cc:1196] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/84d8edc15ea7440c8854cecca42a881f
I20260621 02:43:08.670218 19636 ts_tablet_manager.cc:1939] T 84d8edc15ea7440c8854cecca42a881f P 45bc6b1efa0d4407a2d2c083a5e60457: Deleting consensus metadata
I20260621 02:43:08.670522 19637 tablet_replica.cc:330] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: stopping tablet replica
I20260621 02:43:08.671051 19637 raft_consensus.cc:2229] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:43:08.671720 19637 raft_consensus.cc:2258] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:08.672005 19129 catalog_manager.cc:4841] TS 45bc6b1efa0d4407a2d2c083a5e60457 (127.0.97.3:38527): tablet 84d8edc15ea7440c8854cecca42a881f (table test-table [id=72e7a949909348abb928efe54c4dcb45]) successfully deleted
I20260621 02:43:08.673702 19637 ts_tablet_manager.cc:1905] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:08.677650 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.653735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.0.97.1:35895)
0621 02:43:08.653917 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.653923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.669830 (+ 15907us) server_negotiation.cc:407] Connection header received
0621 02:43:08.670029 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.670036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.670159 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.670266 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:08.670902 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.671972 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.673933 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.674564 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.674698 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.676013 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:08.676024 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:08.676041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:08.676790 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:08.676850 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:08.676874 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:08.676985 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.677157 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.677265 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:43:08.678422 19326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:08.680372 19637 ts_tablet_manager.cc:1918] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:08.680631 19637 log.cc:1196] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/84d8edc15ea7440c8854cecca42a881f
I20260621 02:43:08.681272 19637 ts_tablet_manager.cc:1939] T 84d8edc15ea7440c8854cecca42a881f P 7510f48d39e742a1bc9a7636ce96a4c0: Deleting consensus metadata
I20260621 02:43:08.683106 19131 catalog_manager.cc:4841] TS 7510f48d39e742a1bc9a7636ce96a4c0 (127.0.97.2:44769): tablet 84d8edc15ea7440c8854cecca42a881f (table test-table [id=72e7a949909348abb928efe54c4dcb45]) successfully deleted
I20260621 02:43:08.770102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 19204
I20260621 02:43:08.788306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 19335
I20260621 02:43:08.796332 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.789240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.0.97.2:44769)
0621 02:43:08.789468 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.789476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.789500 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:43:08.789571 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.789578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.789690 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.789816 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:08.790164 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.791379 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.792717 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.793425 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.793550 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.794657 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:08.794668 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:08.794693 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:08.795485 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:08.795543 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:08.795569 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:08.795691 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.795845 (+ 154us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.795942 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260621 02:43:08.797078 19457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:08.916006 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 19335
I20260621 02:43:08.935271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 19465
I20260621 02:43:08.942019 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:08.936027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.0.97.3:38527)
0621 02:43:08.936196 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:08.936200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:08.936219 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:43:08.936359 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:08.936362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:08.936449 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:08.936533 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:08.936904 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.937741 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.938958 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:08.939364 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:08.939454 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:08.940689 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:08.940701 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:08.940715 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:08.941402 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:08.941468 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:08.941489 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:08.941597 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:08.941692 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 02:43:08.941777 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:43:08.942682 19587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:09.031402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 19465
I20260621 02:43:09.047992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 19114
I20260621 02:43:09.049054 19175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:09.158557 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 19114
2026-06-21T02:43:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2753 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:09 dist-test-slave-jkn7 krb5kdc[19645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:09 dist-test-slave-jkn7 krb5kdc[19645](info): set up 2 sockets
Jun 21 02:43:09 dist-test-slave-jkn7 krb5kdc[19645](info): commencing operation
krb5kdc: starting...
W20260621 02:43:10.265601 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:10 dist-test-slave-jkn7 krb5kdc[19645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:10.497824 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45683
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:10.831586 19658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:10.831859 19658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:10.832010 19658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:10.839227 19658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:10.839329 19658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:10.839413 19658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:10.839516 19658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:10.839579 19658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:10.849226 19658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:10.850505 19658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:10.851864 19658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:10.863471 19666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:10.863497 19664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:10.864156 19663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:10.865259 19658 server_base.cc:1034] running on GCE node
I20260621 02:43:10.866050 19658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:10.867429 19658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:10.868664 19658 hybrid_clock.cc:648] HybridClock initialized: now 1782009790868625 us; error 58 us; skew 500 ppm
I20260621 02:43:10.868999 19658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:10 dist-test-slave-jkn7 krb5kdc[19645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:10.876284 19658 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:10.877772 19658 webserver.cc:460] Webserver started at http://127.0.97.62:46865/ using document root <none> and password file <none>
I20260621 02:43:10.878448 19658 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:10.878594 19658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:10.878902 19658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:10.881870 19658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "41bc38aabd68450a908fd80993d3e7e1"
format_stamp: "Formatted at 2026-06-21 02:43:10 on dist-test-slave-jkn7"
server_key: "01bb5732ede27cc13b11826665003de7"
server_key_iv: "50b3e0af9fbb8f456b0cb6619e4c5f19"
server_key_version: "encryptionkey@0"
I20260621 02:43:10.882601 19658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "41bc38aabd68450a908fd80993d3e7e1"
format_stamp: "Formatted at 2026-06-21 02:43:10 on dist-test-slave-jkn7"
server_key: "01bb5732ede27cc13b11826665003de7"
server_key_iv: "50b3e0af9fbb8f456b0cb6619e4c5f19"
server_key_version: "encryptionkey@0"
I20260621 02:43:10.887506 19658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:43:10.890776 19673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:10.892031 19658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:43:10.892210 19658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "41bc38aabd68450a908fd80993d3e7e1"
format_stamp: "Formatted at 2026-06-21 02:43:10 on dist-test-slave-jkn7"
server_key: "01bb5732ede27cc13b11826665003de7"
server_key_iv: "50b3e0af9fbb8f456b0cb6619e4c5f19"
server_key_version: "encryptionkey@0"
I20260621 02:43:10.892395 19658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:10.931874 19658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:10.935768 19658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:10.936054 19658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:10.954157 19658 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45683
I20260621 02:43:10.954200 19724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45683 every 8 connection(s)
I20260621 02:43:10.955427 19658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:10.958887 19725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:10.963495 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 19658
I20260621 02:43:10.963681 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:10.964124 388 external_mini_cluster.cc:1428] Setting key 2b917d18c7c856eb113ba84c4f2a17cd
I20260621 02:43:10.969518 19725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1: Bootstrap starting.
I20260621 02:43:10.972990 19725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:10.974210 19725 log.cc:824] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:10.977524 19725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1: No bootstrap required, opened a new log
Jun 21 02:43:10 dist-test-slave-jkn7 krb5kdc[19645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:10.985388 19725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bc38aabd68450a908fd80993d3e7e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45683 } }
I20260621 02:43:10.985845 19725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:10.985980 19725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41bc38aabd68450a908fd80993d3e7e1, State: Initialized, Role: FOLLOWER
I20260621 02:43:10.986538 19725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bc38aabd68450a908fd80993d3e7e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45683 } }
I20260621 02:43:10.986783 19725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:10.986935 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:10.965891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.0.97.62:45683)
0621 02:43:10.966664 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:10.966675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:10.966713 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:43:10.968447 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:10.968468 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:10.968816 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:10.969112 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:10.970872 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:10.972161 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:10.974125 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:10.974974 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:10.975191 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:10.979557 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:10.979565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:10.979587 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:10.979620 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:10.982736 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:10.983421 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:10.983427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:10.983432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:10.983593 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:10.983992 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:10.983995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:10.983997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:10.984486 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:10.984824 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:10.985237 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 02:43:10.985664 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":382,"threads_started":1}
I20260621 02:43:10.986990 19725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:10.987205 19725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:10.990177 19725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bc38aabd68450a908fd80993d3e7e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45683 } }
I20260621 02:43:10.990648 19725 leader_election.cc:302] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41bc38aabd68450a908fd80993d3e7e1; no voters:
I20260621 02:43:10.991199 19725 leader_election.cc:288] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:10.991320 19730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:10.991995 19730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [term 1 LEADER]: Becoming Leader. State: Replica: 41bc38aabd68450a908fd80993d3e7e1, State: Running, Role: LEADER
I20260621 02:43:10.992560 19730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bc38aabd68450a908fd80993d3e7e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45683 } }
I20260621 02:43:10.993126 19725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:10.996080 19731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41bc38aabd68450a908fd80993d3e7e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bc38aabd68450a908fd80993d3e7e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45683 } } }
I20260621 02:43:10.996454 19731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:10.996814 19732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41bc38aabd68450a908fd80993d3e7e1. Latest consensus state: current_term: 1 leader_uuid: "41bc38aabd68450a908fd80993d3e7e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bc38aabd68450a908fd80993d3e7e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45683 } } }
I20260621 02:43:10.997260 19732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:10.997861 19738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:11.004282 19738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:11.013500 19738 catalog_manager.cc:1347] Generated new cluster ID: 1c8d8c3d371844ee98915af1595d90da
I20260621 02:43:11.013737 19738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:11.036181 19738 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:11.038095 19738 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:11.059809 19738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1: Generated new TSK 0
I20260621 02:43:11.060599 19738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260621 02:43:11.115981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45683
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:11.440027 19751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:11.440291 19751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:11.440383 19751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:11.447355 19751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:11.447461 19751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:11.447569 19751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:43:11.456785 19751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:11.457947 19751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:11.459160 19751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:11.472012 19759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:11.472138 19756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:11.472061 19757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:11.473097 19751 server_base.cc:1034] running on GCE node
I20260621 02:43:11.473672 19751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:11.474495 19751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:11.475714 19751 hybrid_clock.cc:648] HybridClock initialized: now 1782009791475668 us; error 35 us; skew 500 ppm
I20260621 02:43:11.476049 19751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:11 dist-test-slave-jkn7 krb5kdc[19645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:11.479146 19751 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260621 02:43:11.480526 19751 webserver.cc:460] Webserver started at http://127.0.97.1:44889/ using document root <none> and password file <none>
I20260621 02:43:11.481123 19751 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:11.481217 19751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:11.481552 19751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:11.484229 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "fe467593ccdb41ac8b3dea2fae20f354"
format_stamp: "Formatted at 2026-06-21 02:43:11 on dist-test-slave-jkn7"
server_key: "824eabfd0cd9f37dbbab1b50e706251c"
server_key_iv: "eb64f652f6a2526bc25643ce25c05aa0"
server_key_version: "encryptionkey@0"
I20260621 02:43:11.484930 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fe467593ccdb41ac8b3dea2fae20f354"
format_stamp: "Formatted at 2026-06-21 02:43:11 on dist-test-slave-jkn7"
server_key: "824eabfd0cd9f37dbbab1b50e706251c"
server_key_iv: "eb64f652f6a2526bc25643ce25c05aa0"
server_key_version: "encryptionkey@0"
I20260621 02:43:11.490543 19751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:43:11.493793 19766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:11.495150 19751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:43:11.495345 19751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "fe467593ccdb41ac8b3dea2fae20f354"
format_stamp: "Formatted at 2026-06-21 02:43:11 on dist-test-slave-jkn7"
server_key: "824eabfd0cd9f37dbbab1b50e706251c"
server_key_iv: "eb64f652f6a2526bc25643ce25c05aa0"
server_key_version: "encryptionkey@0"
I20260621 02:43:11.495543 19751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:11.509773 19751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:11.513631 19751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:11.513908 19751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:11.514814 19751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:11.516040 19751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:11.516112 19751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:11.516276 19751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:11.516340 19751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:11.548372 19751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34169
I20260621 02:43:11.548391 19878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34169 every 8 connection(s)
I20260621 02:43:11.549655 19751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:43:11.558604 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 19751
I20260621 02:43:11.558792 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:43:11.559291 388 external_mini_cluster.cc:1428] Setting key a86481d726f3d9579181317acd2c0f36
Jun 21 02:43:11 dist-test-slave-jkn7 krb5kdc[19645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:11.570832 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:11.552070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49855 (local address 127.0.97.62:45683)
0621 02:43:11.552506 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:11.552511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:11.553429 (+ 918us) server_negotiation.cc:407] Connection header received
0621 02:43:11.555187 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:11.555192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:11.555287 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:11.555405 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:11.556797 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:11.557578 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:11.559409 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:11.559979 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:11.560142 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:11.564125 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:11.564135 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:11.564142 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:11.564189 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:11.567856 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:11.568572 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:11.568580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:11.568587 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:11.568735 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:11.569123 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:11.569131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:11.569138 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:11.569571 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:11.569720 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:11.570380 (+ 660us) server_negotiation.cc:299] Negotiation successful
0621 02:43:11.570570 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":222,"threads_started":1}
I20260621 02:43:11.572268 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:11.552561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45683 (local address 127.0.97.1:49855)
0621 02:43:11.553228 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:11.553263 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:11.554926 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:11.555633 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:11.555642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:11.556111 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:11.556628 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:11.556643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:11.557744 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:11.557748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:11.559245 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:11.559260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:11.560225 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:11.560232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:11.560480 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:11.561229 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:11.561261 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:11.563893 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:11.568004 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:11.568012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:11.568026 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:11.568444 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:11.568842 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:11.568845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:11.568847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:11.569017 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:11.569702 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:11.569708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:11.570028 (+ 320us) client_negotiation.cc:769] Sending connection context
0621 02:43:11.570246 (+ 218us) client_negotiation.cc:240] Negotiation successful
0621 02:43:11.570866 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":259,"threads_started":1}
I20260621 02:43:11.573884 19879 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45683
I20260621 02:43:11.574191 19879 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:11.575033 19879 heartbeater.cc:511] Master 127.0.97.62:45683 requested a full tablet report, sending...
I20260621 02:43:11.577332 19690 ts_manager.cc:194] Registered new tserver with Master: fe467593ccdb41ac8b3dea2fae20f354 (127.0.97.1:34169)
I20260621 02:43:11.579480 19690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49855
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260621 02:43:11.620021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45683
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:11.939901 19885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:11.940274 19885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:11.940454 19885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:11.947868 19885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:11.948065 19885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:11.948267 19885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:43:11.961362 19885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:11.962812 19885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:11.964416 19885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:11.979012 19891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:11.978945 19890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:11.979046 19893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:11.979595 19885 server_base.cc:1034] running on GCE node
I20260621 02:43:11.980144 19885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:11.980959 19885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:11.982191 19885 hybrid_clock.cc:648] HybridClock initialized: now 1782009791982140 us; error 63 us; skew 500 ppm
I20260621 02:43:11.982503 19885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:11 dist-test-slave-jkn7 krb5kdc[19645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:11.985574 19885 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260621 02:43:11.986974 19885 webserver.cc:460] Webserver started at http://127.0.97.2:42111/ using document root <none> and password file <none>
I20260621 02:43:11.987649 19885 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:11.987792 19885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:11.988098 19885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:11.990916 19885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "b8c628777c194531b6c7b67bd2a6a171"
format_stamp: "Formatted at 2026-06-21 02:43:11 on dist-test-slave-jkn7"
server_key: "032e121a067bdc997a58e80a9610d5a1"
server_key_iv: "1964312ca7f777846a0b508fc44eb9ef"
server_key_version: "encryptionkey@0"
I20260621 02:43:11.991699 19885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b8c628777c194531b6c7b67bd2a6a171"
format_stamp: "Formatted at 2026-06-21 02:43:11 on dist-test-slave-jkn7"
server_key: "032e121a067bdc997a58e80a9610d5a1"
server_key_iv: "1964312ca7f777846a0b508fc44eb9ef"
server_key_version: "encryptionkey@0"
I20260621 02:43:11.996830 19885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:43:12.000630 19900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:12.002133 19885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:43:12.002321 19885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "b8c628777c194531b6c7b67bd2a6a171"
format_stamp: "Formatted at 2026-06-21 02:43:11 on dist-test-slave-jkn7"
server_key: "032e121a067bdc997a58e80a9610d5a1"
server_key_iv: "1964312ca7f777846a0b508fc44eb9ef"
server_key_version: "encryptionkey@0"
I20260621 02:43:12.002529 19885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:12.021051 19885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:12.025148 19885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:12.025460 19885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:12.026418 19885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:12.027758 19885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:12.027863 19885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:12.028016 19885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:12.028087 19885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:12.061528 19885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34763
I20260621 02:43:12.061612 20012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34763 every 8 connection(s)
I20260621 02:43:12.062732 19885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:43:12.072814 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 19885
I20260621 02:43:12.073031 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:43:12.073611 388 external_mini_cluster.cc:1428] Setting key 290438302c51f6b35072c220bc3aff8b
Jun 21 02:43:12 dist-test-slave-jkn7 krb5kdc[19645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:12.083940 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.065216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39693 (local address 127.0.97.62:45683)
0621 02:43:12.065502 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.065509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.066577 (+ 1068us) server_negotiation.cc:407] Connection header received
0621 02:43:12.068469 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.068474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.068566 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.068676 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:12.070146 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.070829 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.072316 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.072767 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.072905 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.076620 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:12.076629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:12.076637 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:12.076682 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:12.080567 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:12.081339 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:12.081347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:12.081353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:12.081511 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:12.081983 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:12.081991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:12.081997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:12.082355 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:12.082517 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.083225 (+ 708us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.083480 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260621 02:43:12.084611 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.065715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45683 (local address 127.0.97.2:39693)
0621 02:43:12.066405 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:12.066447 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:12.068201 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:12.068901 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:12.068909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:12.069451 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:12.069981 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.069997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.070968 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.070976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.072151 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.072162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.073020 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.073030 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.073352 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.074068 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:12.074103 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:12.076391 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:12.080705 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:12.080711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:12.080724 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:12.081148 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:12.081653 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:12.081657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:12.081659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:12.081852 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:12.082498 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:12.082505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:12.082836 (+ 331us) client_negotiation.cc:769] Sending connection context
0621 02:43:12.083072 (+ 236us) client_negotiation.cc:240] Negotiation successful
0621 02:43:12.083467 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":279,"threads_started":1}
I20260621 02:43:12.086112 20013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45683
I20260621 02:43:12.086431 20013 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:12.087106 20013 heartbeater.cc:511] Master 127.0.97.62:45683 requested a full tablet report, sending...
I20260621 02:43:12.088668 19690 ts_manager.cc:194] Registered new tserver with Master: b8c628777c194531b6c7b67bd2a6a171 (127.0.97.2:34763)
I20260621 02:43:12.089926 19690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39693
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260621 02:43:12.130614 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45683
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:12.454926 20018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:12.455258 20018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:12.455425 20018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:12.462865 20018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:12.462980 20018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:12.463160 20018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:43:12.473084 20018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:12.474357 20018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:12.475601 20018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:12.487857 20024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:12.488060 20026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:12.488327 20023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:12.488723 20018 server_base.cc:1034] running on GCE node
I20260621 02:43:12.489277 20018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:12.490279 20018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:12.491513 20018 hybrid_clock.cc:648] HybridClock initialized: now 1782009792491460 us; error 34 us; skew 500 ppm
I20260621 02:43:12.491845 20018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:12 dist-test-slave-jkn7 krb5kdc[19645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:12.495244 20018 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260621 02:43:12.496843 20018 webserver.cc:460] Webserver started at http://127.0.97.3:35215/ using document root <none> and password file <none>
I20260621 02:43:12.497630 20018 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:12.497756 20018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:12.498075 20018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:12.501147 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "e183e089551a4401993d50ff4db1f84b"
format_stamp: "Formatted at 2026-06-21 02:43:12 on dist-test-slave-jkn7"
server_key: "324cedbab73e7fc7146ad37b45ff7ae8"
server_key_iv: "6dd2e01a252e5788c6dc8718777d48a1"
server_key_version: "encryptionkey@0"
I20260621 02:43:12.501961 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e183e089551a4401993d50ff4db1f84b"
format_stamp: "Formatted at 2026-06-21 02:43:12 on dist-test-slave-jkn7"
server_key: "324cedbab73e7fc7146ad37b45ff7ae8"
server_key_iv: "6dd2e01a252e5788c6dc8718777d48a1"
server_key_version: "encryptionkey@0"
I20260621 02:43:12.506822 20018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:43:12.510310 20033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:12.511686 20018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:43:12.511869 20018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "e183e089551a4401993d50ff4db1f84b"
format_stamp: "Formatted at 2026-06-21 02:43:12 on dist-test-slave-jkn7"
server_key: "324cedbab73e7fc7146ad37b45ff7ae8"
server_key_iv: "6dd2e01a252e5788c6dc8718777d48a1"
server_key_version: "encryptionkey@0"
I20260621 02:43:12.512101 20018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:12.538218 20018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:12.542096 20018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:12.542383 20018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:12.543399 20018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:12.544709 20018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:12.544816 20018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:12.544970 20018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:12.545045 20018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:12.578625 20018 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42719
I20260621 02:43:12.578711 20145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42719 every 8 connection(s)
I20260621 02:43:12.579861 20018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:43:12.583035 19879 heartbeater.cc:503] Master 127.0.97.62:45683 was elected leader, sending a full tablet report...
I20260621 02:43:12.583928 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 20018
I20260621 02:43:12.584107 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:43:12.584682 388 external_mini_cluster.cc:1428] Setting key 1866c7909d1455ed3e40f9516fd550c2
Jun 21 02:43:12 dist-test-slave-jkn7 krb5kdc[19645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:12.600096 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.582554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47383 (local address 127.0.97.62:45683)
0621 02:43:12.582768 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.582773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.583961 (+ 1188us) server_negotiation.cc:407] Connection header received
0621 02:43:12.586221 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.586233 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.586295 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.586379 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:12.587764 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.588529 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.590074 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.590456 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.590558 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.594017 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:12.594022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:12.594026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:12.594057 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:12.596675 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:12.597449 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:12.597454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:12.597456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:12.597589 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:12.598011 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:12.598019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:12.598024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:12.598380 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:12.598562 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.599443 (+ 881us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.599672 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:43:12.601064 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.582973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45683 (local address 127.0.97.3:47383)
0621 02:43:12.583802 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:12.583855 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:12.585971 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:12.586581 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:12.586588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:12.587064 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:12.587590 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.587606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.588661 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.588664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.589921 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.589936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.590645 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.590650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.590840 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.591479 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:12.591502 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:12.593764 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:12.596817 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:12.596823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:12.596852 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:12.597267 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:12.597691 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:12.597694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:12.597696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:12.597872 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:12.598543 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:12.598552 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:12.598934 (+ 382us) client_negotiation.cc:769] Sending connection context
0621 02:43:12.599300 (+ 366us) client_negotiation.cc:240] Negotiation successful
0621 02:43:12.599825 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":390,"threads_started":1}
I20260621 02:43:12.602650 20146 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45683
I20260621 02:43:12.602984 20146 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:12.603691 20146 heartbeater.cc:511] Master 127.0.97.62:45683 requested a full tablet report, sending...
I20260621 02:43:12.605216 19690 ts_manager.cc:194] Registered new tserver with Master: e183e089551a4401993d50ff4db1f84b (127.0.97.3:42719)
I20260621 02:43:12.606511 19690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47383
I20260621 02:43:12.612488 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:43:12.629751 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.616770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.0.97.62:45683)
0621 02:43:12.617002 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.617011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.617348 (+ 337us) server_negotiation.cc:407] Connection header received
0621 02:43:12.617550 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.617556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.617635 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.617740 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:12.619141 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.619870 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.621143 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.621650 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.621789 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.624185 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:12.624191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:12.624198 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:12.624238 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:12.626813 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:12.627717 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:12.627723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:12.627727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:12.627921 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:12.628406 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:12.628411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:12.628415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:12.628746 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:12.628936 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.629333 (+ 397us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.629509 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:43:12.633486 19690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:43:12.640196 19690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:43:12.662855 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.652446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.0.97.3:42719)
0621 02:43:12.654701 (+ 2255us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.654710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.654743 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:43:12.654844 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.654851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.655067 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.656211 (+ 1144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:12.656510 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.658078 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.660401 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.661525 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.661683 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.661908 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.662358 (+ 450us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.662522 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2067,"thread_start_us":1906,"threads_started":1}
I20260621 02:43:12.662880 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.651828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42719 (local address 127.0.0.1:50066)
0621 02:43:12.653369 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:12.653387 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:12.653483 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:12.655281 (+ 1798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:12.655288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:12.655313 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:12.655998 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.656010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.658216 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.658225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.660184 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.660196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.661722 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.661728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.661913 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.662013 (+ 100us) client_negotiation.cc:769] Sending connection context
0621 02:43:12.662167 (+ 154us) client_negotiation.cc:240] Negotiation successful
0621 02:43:12.662342 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"mutex_wait_us":240,"spinlock_wait_cycles":2432,"thread_start_us":1059,"threads_started":1}
I20260621 02:43:12.663017 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.651657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34169 (local address 127.0.0.1:60224)
0621 02:43:12.652573 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:12.652612 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:12.652786 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:12.655234 (+ 2448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:12.655241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:12.655269 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:12.655737 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.655750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.657728 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.657736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.660105 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.660124 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.662049 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.662056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.662229 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.662271 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 02:43:12.662340 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 02:43:12.662458 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":3,"spinlock_wait_cycles":42368,"thread_start_us":1420,"threads_started":1}
I20260621 02:43:12.663578 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.653728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34763 (local address 127.0.0.1:51682)
0621 02:43:12.654327 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:12.654353 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:12.654469 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:12.655445 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:12.655449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:12.655484 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:12.656004 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.656012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.658083 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.658088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.660334 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.660349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.662143 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.662152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.662322 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.662350 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 02:43:12.662432 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 02:43:12.662575 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":1,"thread_start_us":322,"threads_started":1}
I20260621 02:43:12.663623 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.654205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.0.97.2:34763)
0621 02:43:12.654924 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.654931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.654965 (+ 34us) server_negotiation.cc:407] Connection header received
0621 02:43:12.655057 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.655063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.655250 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.656088 (+ 838us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:12.656311 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.657938 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.660518 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.661940 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.662060 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.662240 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.663135 (+ 895us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.663284 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":437,"threads_started":1}
I20260621 02:43:12.664233 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.651913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.0.97.1:34169)
0621 02:43:12.654698 (+ 2785us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.654707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.654737 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:43:12.654830 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.654837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.655031 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.655221 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:12.655922 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.657605 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.660299 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.661791 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.663493 (+ 1702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.663700 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.663763 (+ 63us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.663922 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1959,"thread_start_us":1762,"threads_started":1}
I20260621 02:43:12.666998 20081 tablet_service.cc:1467] Processing CreateTablet for tablet cccf812d2e8b40b8b790d52bff1ec736 (DEFAULT_TABLE table=test-table [id=20593cd5b0804455b12ee40ed2794aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:12.667289 19814 tablet_service.cc:1467] Processing CreateTablet for tablet cccf812d2e8b40b8b790d52bff1ec736 (DEFAULT_TABLE table=test-table [id=20593cd5b0804455b12ee40ed2794aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:12.667565 19948 tablet_service.cc:1467] Processing CreateTablet for tablet cccf812d2e8b40b8b790d52bff1ec736 (DEFAULT_TABLE table=test-table [id=20593cd5b0804455b12ee40ed2794aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:12.668547 19814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf812d2e8b40b8b790d52bff1ec736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:12.668553 20081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf812d2e8b40b8b790d52bff1ec736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:12.668784 19948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf812d2e8b40b8b790d52bff1ec736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:12.680474 20164 tablet_bootstrap.cc:492] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: Bootstrap starting.
I20260621 02:43:12.683593 20166 tablet_bootstrap.cc:492] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: Bootstrap starting.
I20260621 02:43:12.684739 20165 tablet_bootstrap.cc:492] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: Bootstrap starting.
I20260621 02:43:12.686179 20164 tablet_bootstrap.cc:654] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:12.687510 20164 log.cc:824] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:12.687561 20165 tablet_bootstrap.cc:654] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:12.687608 20166 tablet_bootstrap.cc:654] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:12.688697 20166 log.cc:824] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:12.688777 20165 log.cc:824] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:12.690898 20164 tablet_bootstrap.cc:492] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: No bootstrap required, opened a new log
I20260621 02:43:12.691192 20164 ts_tablet_manager.cc:1397] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260621 02:43:12.691674 20166 tablet_bootstrap.cc:492] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: No bootstrap required, opened a new log
I20260621 02:43:12.691931 20166 ts_tablet_manager.cc:1397] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260621 02:43:12.692515 20165 tablet_bootstrap.cc:492] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: No bootstrap required, opened a new log
I20260621 02:43:12.692807 20165 ts_tablet_manager.cc:1397] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260621 02:43:12.700062 20166 raft_consensus.cc:348] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } }
I20260621 02:43:12.700456 20166 raft_consensus.cc:374] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:12.700562 20166 raft_consensus.cc:729] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c628777c194531b6c7b67bd2a6a171, State: Initialized, Role: FOLLOWER
I20260621 02:43:12.701220 20166 consensus_queue.cc:260] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } }
I20260621 02:43:12.702520 20013 heartbeater.cc:503] Master 127.0.97.62:45683 was elected leader, sending a full tablet report...
I20260621 02:43:12.702292 20164 raft_consensus.cc:348] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } }
I20260621 02:43:12.702740 20164 raft_consensus.cc:374] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:12.702832 20164 raft_consensus.cc:729] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe467593ccdb41ac8b3dea2fae20f354, State: Initialized, Role: FOLLOWER
I20260621 02:43:12.703256 20166 ts_tablet_manager.cc:1428] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260621 02:43:12.703572 20164 consensus_queue.cc:260] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } }
I20260621 02:43:12.703487 20165 raft_consensus.cc:348] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } }
I20260621 02:43:12.703899 20165 raft_consensus.cc:374] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:12.704010 20165 raft_consensus.cc:729] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e183e089551a4401993d50ff4db1f84b, State: Initialized, Role: FOLLOWER
I20260621 02:43:12.704802 20165 consensus_queue.cc:260] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } }
I20260621 02:43:12.709175 20146 heartbeater.cc:503] Master 127.0.97.62:45683 was elected leader, sending a full tablet report...
I20260621 02:43:12.710326 20165 ts_tablet_manager.cc:1428] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20260621 02:43:12.711903 20164 ts_tablet_manager.cc:1428] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
W20260621 02:43:12.802033 19880 tablet.cc:2369] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:43:12.814803 20014 tablet.cc:2369] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:43:12.831517 20147 tablet.cc:2369] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:43:12.838078 20171 raft_consensus.cc:482] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:43:12.838380 20171 raft_consensus.cc:504] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } }
I20260621 02:43:12.840042 20171 leader_election.cc:288] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e183e089551a4401993d50ff4db1f84b (127.0.97.3:42719), b8c628777c194531b6c7b67bd2a6a171 (127.0.97.2:34763)
I20260621 02:43:12.848479 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.840665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44593 (local address 127.0.97.3:42719)
0621 02:43:12.840936 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.840946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.841787 (+ 841us) server_negotiation.cc:407] Connection header received
0621 02:43:12.841950 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.841957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.842051 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.842134 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:12.842808 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.843664 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.846104 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.847339 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.847487 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.847594 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.848080 (+ 486us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.848217 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260621 02:43:12.848479 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.840617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44251 (local address 127.0.97.2:34763)
0621 02:43:12.840863 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.840873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.841892 (+ 1019us) server_negotiation.cc:407] Connection header received
0621 02:43:12.842067 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.842071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.842163 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.842255 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:12.842928 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.843622 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.846183 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.847384 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.847506 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.847600 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.848080 (+ 480us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.848217 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:43:12.848701 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.840605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34763 (local address 127.0.97.1:44251)
0621 02:43:12.841760 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:12.841782 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:12.841929 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:12.842302 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:12.842307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:12.842357 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:12.842764 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.842777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.843766 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.843773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.846032 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.846050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.847672 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.847678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.847908 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.847942 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:43:12.848022 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 02:43:12.848169 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":137,"thread_start_us":510,"threads_started":1}
I20260621 02:43:12.848722 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.840541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42719 (local address 127.0.97.1:44593)
0621 02:43:12.841653 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:12.841670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:12.841803 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:12.842180 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:12.842187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:12.842217 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:12.842639 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.842650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.843832 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.843835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.845939 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:12.845957 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.847637 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.847646 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:12.847850 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.847889 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:43:12.848000 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 02:43:12.848169 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":123,"thread_start_us":795,"threads_started":1}
I20260621 02:43:12.849905 19968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf812d2e8b40b8b790d52bff1ec736" candidate_uuid: "fe467593ccdb41ac8b3dea2fae20f354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c628777c194531b6c7b67bd2a6a171" is_pre_election: true
I20260621 02:43:12.850122 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf812d2e8b40b8b790d52bff1ec736" candidate_uuid: "fe467593ccdb41ac8b3dea2fae20f354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e183e089551a4401993d50ff4db1f84b" is_pre_election: true
I20260621 02:43:12.850553 19968 raft_consensus.cc:2454] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe467593ccdb41ac8b3dea2fae20f354 in term 0.
I20260621 02:43:12.850608 20101 raft_consensus.cc:2454] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe467593ccdb41ac8b3dea2fae20f354 in term 0.
I20260621 02:43:12.851265 19769 leader_election.cc:302] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e183e089551a4401993d50ff4db1f84b, fe467593ccdb41ac8b3dea2fae20f354; no voters:
I20260621 02:43:12.851671 20171 raft_consensus.cc:2781] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:43:12.851805 20171 raft_consensus.cc:482] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:43:12.851949 20171 raft_consensus.cc:3037] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:12.854239 20171 raft_consensus.cc:504] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } }
I20260621 02:43:12.854835 20171 leader_election.cc:288] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [CANDIDATE]: Term 1 election: Requested vote from peers e183e089551a4401993d50ff4db1f84b (127.0.97.3:42719), b8c628777c194531b6c7b67bd2a6a171 (127.0.97.2:34763)
I20260621 02:43:12.855324 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf812d2e8b40b8b790d52bff1ec736" candidate_uuid: "fe467593ccdb41ac8b3dea2fae20f354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e183e089551a4401993d50ff4db1f84b"
I20260621 02:43:12.855322 19968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf812d2e8b40b8b790d52bff1ec736" candidate_uuid: "fe467593ccdb41ac8b3dea2fae20f354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c628777c194531b6c7b67bd2a6a171"
I20260621 02:43:12.855540 19968 raft_consensus.cc:3037] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:12.855521 20101 raft_consensus.cc:3037] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:12.857744 19968 raft_consensus.cc:2454] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe467593ccdb41ac8b3dea2fae20f354 in term 1.
I20260621 02:43:12.857779 20101 raft_consensus.cc:2454] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe467593ccdb41ac8b3dea2fae20f354 in term 1.
I20260621 02:43:12.858182 19768 leader_election.cc:302] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8c628777c194531b6c7b67bd2a6a171, fe467593ccdb41ac8b3dea2fae20f354; no voters:
I20260621 02:43:12.858502 20171 raft_consensus.cc:2781] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:12.858970 20171 raft_consensus.cc:686] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 1 LEADER]: Becoming Leader. State: Replica: fe467593ccdb41ac8b3dea2fae20f354, State: Running, Role: LEADER
I20260621 02:43:12.859516 20171 consensus_queue.cc:237] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } }
I20260621 02:43:12.863973 19689 catalog_manager.cc:5495] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe467593ccdb41ac8b3dea2fae20f354 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fe467593ccdb41ac8b3dea2fae20f354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe467593ccdb41ac8b3dea2fae20f354" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 } health_report { overall_health: UNKNOWN } } }
I20260621 02:43:12.904222 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.898785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.0.97.1:34169)
0621 02:43:12.899056 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.899064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.899086 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:43:12.899288 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.899297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.899423 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.899554 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:12.899852 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.900674 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.902219 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.902701 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.902801 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.903245 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:12.903759 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:12.903872 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.903957 (+ 85us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.904024 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260621 02:43:12.912475 20101 raft_consensus.cc:1261] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [term 1 FOLLOWER]: Refusing update from remote peer fe467593ccdb41ac8b3dea2fae20f354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:12.912472 19968 raft_consensus.cc:1261] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [term 1 FOLLOWER]: Refusing update from remote peer fe467593ccdb41ac8b3dea2fae20f354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:12.913481 20175 consensus_queue.cc:1035] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e183e089551a4401993d50ff4db1f84b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:12.913868 20171 consensus_queue.cc:1035] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8c628777c194531b6c7b67bd2a6a171" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:12.925966 20180 mvcc.cc:205] Tried to move back new op lower bound from 7299112111760056320 to 7299112111556128768. Current Snapshot: MvccSnapshot[applied={T|T < 7299112111760056320}]
I20260621 02:43:12.945942 19690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:12.946216 19690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:12.949540 19690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41bc38aabd68450a908fd80993d3e7e1: Sending DeleteTablet for 3 replicas of tablet cccf812d2e8b40b8b790d52bff1ec736
I20260621 02:43:12.950664 20081 tablet_service.cc:1514] Processing DeleteTablet for tablet cccf812d2e8b40b8b790d52bff1ec736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50066
I20260621 02:43:12.950802 19814 tablet_service.cc:1514] Processing DeleteTablet for tablet cccf812d2e8b40b8b790d52bff1ec736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60224
I20260621 02:43:12.950906 19948 tablet_service.cc:1514] Processing DeleteTablet for tablet cccf812d2e8b40b8b790d52bff1ec736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51682
I20260621 02:43:12.951995 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 19751
I20260621 02:43:12.964228 20188 tablet_replica.cc:330] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: stopping tablet replica
I20260621 02:43:12.964838 20188 raft_consensus.cc:2229] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:43:12.965860 20190 tablet_replica.cc:330] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: stopping tablet replica
I20260621 02:43:12.966377 20190 raft_consensus.cc:2229] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:12.967046 20190 raft_consensus.cc:2258] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:12.967893 20188 raft_consensus.cc:2258] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:12.971014 20190 ts_tablet_manager.cc:1905] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:12.975407 20189 tablet_replica.cc:330] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: stopping tablet replica
I20260621 02:43:12.975850 20189 raft_consensus.cc:2229] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:12.976298 20188 ts_tablet_manager.cc:1905] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:12.976361 20189 raft_consensus.cc:2258] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:12.978407 20189 ts_tablet_manager.cc:1905] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:12.983733 20188 ts_tablet_manager.cc:1918] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:12.983907 20188 log.cc:1196] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-0/wal/wals/cccf812d2e8b40b8b790d52bff1ec736
I20260621 02:43:12.986720 20190 ts_tablet_manager.cc:1918] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:12.986896 20190 log.cc:1196] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/cccf812d2e8b40b8b790d52bff1ec736
I20260621 02:43:12.986989 20188 ts_tablet_manager.cc:1939] T cccf812d2e8b40b8b790d52bff1ec736 P fe467593ccdb41ac8b3dea2fae20f354: Deleting consensus metadata
I20260621 02:43:12.987454 20190 ts_tablet_manager.cc:1939] T cccf812d2e8b40b8b790d52bff1ec736 P b8c628777c194531b6c7b67bd2a6a171: Deleting consensus metadata
I20260621 02:43:12.988379 20189 ts_tablet_manager.cc:1918] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:12.988536 20189 log.cc:1196] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/cccf812d2e8b40b8b790d52bff1ec736
I20260621 02:43:12.988890 19675 catalog_manager.cc:4841] TS fe467593ccdb41ac8b3dea2fae20f354 (127.0.97.1:34169): tablet cccf812d2e8b40b8b790d52bff1ec736 (table test-table [id=20593cd5b0804455b12ee40ed2794aff]) successfully deleted
I20260621 02:43:12.989065 20189 ts_tablet_manager.cc:1939] T cccf812d2e8b40b8b790d52bff1ec736 P e183e089551a4401993d50ff4db1f84b: Deleting consensus metadata
I20260621 02:43:12.989214 19675 catalog_manager.cc:4841] TS b8c628777c194531b6c7b67bd2a6a171 (127.0.97.2:34763): tablet cccf812d2e8b40b8b790d52bff1ec736 (table test-table [id=20593cd5b0804455b12ee40ed2794aff]) successfully deleted
I20260621 02:43:12.990392 19676 catalog_manager.cc:4841] TS e183e089551a4401993d50ff4db1f84b (127.0.97.3:42719): tablet cccf812d2e8b40b8b790d52bff1ec736 (table test-table [id=20593cd5b0804455b12ee40ed2794aff]) successfully deleted
Jun 21 02:43:12 dist-test-slave-jkn7 krb5kdc[19645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260621 02:43:12.999517 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:12.953186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.0.97.1:34169)
0621 02:43:12.953380 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:12.953386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:12.959658 (+ 6272us) server_negotiation.cc:407] Connection header received
0621 02:43:12.960355 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:12.960361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:12.960448 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:12.960550 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:12.981466 (+ 20916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.982796 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.984772 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:12.985529 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:12.985641 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:12.993128 (+ 7487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:12.993139 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:12.993153 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:12.993189 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:12.996512 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:12.997327 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:12.997337 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:12.997343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:12.997519 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:12.997923 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:12.997927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:12.997931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:12.998272 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:12.998623 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:12.998899 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 02:43:12.999139 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:43:13.000192 19874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:13.106137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 19751
I20260621 02:43:13.121176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 19885
Jun 21 02:43:13 dist-test-slave-jkn7 krb5kdc[19645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260621 02:43:13.135761 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:13.121949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.0.97.2:34763)
0621 02:43:13.122129 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:13.122134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:13.122151 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:43:13.122296 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:13.122299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:13.122379 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:13.122458 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:13.123645 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:13.124385 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:13.125772 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:13.126155 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:13.126256 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:13.130282 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:13.130305 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:13.130321 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:13.130362 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:13.133234 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:13.133963 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:13.133970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:13.133972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:13.134106 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:13.134425 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:13.134428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:13.134430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:13.134788 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:13.135001 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:13.135219 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 02:43:13.135371 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:43:13.136396 20008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:13.225605 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 19885
I20260621 02:43:13.242085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20018
Jun 21 02:43:13 dist-test-slave-jkn7 krb5kdc[19645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260621 02:43:13.256879 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:13.242864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.0.97.3:42719)
0621 02:43:13.243066 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:13.243074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:13.243099 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:43:13.243325 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:13.243330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:13.243405 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:13.243518 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:13.244860 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:13.245672 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:13.246977 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:13.247367 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:13.247508 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:13.251713 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:13.251727 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:13.251743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:13.251773 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:13.254544 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:13.255111 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:13.255118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:13.255120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:13.255249 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:13.255585 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:13.255588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:13.255590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:13.255917 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:13.256127 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:13.256355 (+ 228us) server_negotiation.cc:299] Negotiation successful
0621 02:43:13.256500 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:43:13.257682 20141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:13.344929 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 20018
I20260621 02:43:13.362125 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 19658
I20260621 02:43:13.363370 19720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:13.472093 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 19658
2026-06-21T02:43:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4313 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T02:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:13Z Disabled control of system clock
I20260621 02:43:13.546294 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41885
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:43:13.869968 20199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:13.870261 20199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:13.870347 20199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:13.877753 20199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:13.877868 20199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:13.877907 20199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:13.877965 20199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:13.878011 20199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:13.887456 20199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:13.888778 20199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:13.890578 20199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:13.902477 20204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:13.902515 20205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:13.902557 20207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:13.903494 20199 server_base.cc:1034] running on GCE node
I20260621 02:43:13.904376 20199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:13.906003 20199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:13.907233 20199 hybrid_clock.cc:648] HybridClock initialized: now 1782009793907190 us; error 52 us; skew 500 ppm
I20260621 02:43:13.907593 20199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:13.909873 20199 webserver.cc:460] Webserver started at http://127.0.97.62:46125/ using document root <none> and password file <none>
I20260621 02:43:13.910560 20199 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:13.910691 20199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:13.911006 20199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:13.913952 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a"
format_stamp: "Formatted at 2026-06-21 02:43:13 on dist-test-slave-jkn7"
server_key: "9a0a822db61b5d2d2e9b60453cac677f"
server_key_iv: "2ced08ee2d1281a4047a5032295e1b38"
server_key_version: "encryptionkey@0"
I20260621 02:43:13.914748 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a"
format_stamp: "Formatted at 2026-06-21 02:43:13 on dist-test-slave-jkn7"
server_key: "9a0a822db61b5d2d2e9b60453cac677f"
server_key_iv: "2ced08ee2d1281a4047a5032295e1b38"
server_key_version: "encryptionkey@0"
I20260621 02:43:13.919713 20199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:43:13.922964 20213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:13.924186 20199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:43:13.924371 20199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a"
format_stamp: "Formatted at 2026-06-21 02:43:13 on dist-test-slave-jkn7"
server_key: "9a0a822db61b5d2d2e9b60453cac677f"
server_key_iv: "2ced08ee2d1281a4047a5032295e1b38"
server_key_version: "encryptionkey@0"
I20260621 02:43:13.924577 20199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:13.942958 20199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:13.943728 20199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:13.944021 20199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:13.962309 20199 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41885
I20260621 02:43:13.962342 20264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41885 every 8 connection(s)
I20260621 02:43:13.963605 20199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:13.967077 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 20199
I20260621 02:43:13.967177 20265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:13.967267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:13.967697 388 external_mini_cluster.cc:1428] Setting key b020a8079c31770704b14a6f16864d55
I20260621 02:43:13.980362 20265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a: Bootstrap starting.
I20260621 02:43:13.981906 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:13.969533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.0.97.62:41885)
0621 02:43:13.970640 (+ 1107us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:13.970650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:13.970689 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:43:13.973046 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:13.973067 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:13.973595 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:13.973903 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:13.974461 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:13.975652 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:13.977589 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:13.978260 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:13.978395 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:13.979619 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:13.979625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:13.979638 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:13.980156 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:13.980227 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:13.980247 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:13.980330 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:13.980605 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 02:43:13.980942 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":457,"threads_started":1}
I20260621 02:43:13.983610 20265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:13.984617 20265 log.cc:824] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:13.987155 20265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a: No bootstrap required, opened a new log
I20260621 02:43:13.994335 20265 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } }
I20260621 02:43:13.994722 20265 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:13.994827 20265 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f92ba3542a410e8d4cbfaf7c7fdd3a, State: Initialized, Role: FOLLOWER
I20260621 02:43:13.995376 20265 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } }
I20260621 02:43:13.995609 20265 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:13.995764 20265 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:13.995936 20265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:13.998137 20265 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } }
I20260621 02:43:13.998651 20265 leader_election.cc:302] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f92ba3542a410e8d4cbfaf7c7fdd3a; no voters:
I20260621 02:43:13.999215 20265 leader_election.cc:288] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:13.999374 20270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:14.000169 20270 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [term 1 LEADER]: Becoming Leader. State: Replica: b3f92ba3542a410e8d4cbfaf7c7fdd3a, State: Running, Role: LEADER
I20260621 02:43:14.000665 20270 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } }
I20260621 02:43:14.001405 20265 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:14.003870 20272 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f92ba3542a410e8d4cbfaf7c7fdd3a. Latest consensus state: current_term: 1 leader_uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } } }
I20260621 02:43:14.004246 20272 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:14.003984 20271 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f92ba3542a410e8d4cbfaf7c7fdd3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } } }
I20260621 02:43:14.004384 20271 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:14.005967 20278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:14.010135 20278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:14.018739 20278 catalog_manager.cc:1347] Generated new cluster ID: 2b97e0876d7344fea7c64a30b73ad66c
I20260621 02:43:14.018857 20278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:14.032792 20278 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:14.034039 20278 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:14.041800 20278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a: Generated new TSK 0
I20260621 02:43:14.042837 20278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:14.051192 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--builtin_ntp_servers=127.0.97.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:43:14.374243 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:14.374541 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:14.374703 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:14.381745 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:14.381876 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:14.382051 20289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260621 02:43:14.390733 20289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:14.391942 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:14.393664 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:14.406399 20294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:14.406486 20295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:14.406432 20297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:14.407552 20289 server_base.cc:1034] running on GCE node
I20260621 02:43:14.408138 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:14.409085 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:14.410287 20289 hybrid_clock.cc:648] HybridClock initialized: now 1782009794410220 us; error 65 us; skew 500 ppm
I20260621 02:43:14.410614 20289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:14.412938 20289 webserver.cc:460] Webserver started at http://127.0.97.1:34789/ using document root <none> and password file <none>
I20260621 02:43:14.413702 20289 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:14.413789 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:14.414067 20289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:14.416885 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/data/instance:
uuid: "1cca920b7a6644b8acb282473336d0bc"
format_stamp: "Formatted at 2026-06-21 02:43:14 on dist-test-slave-jkn7"
server_key: "4580ba97cc4ceba1123f7baf4ba9d47a"
server_key_iv: "465e60e2bc872bd97591d91d193512fd"
server_key_version: "encryptionkey@0"
I20260621 02:43:14.417697 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1cca920b7a6644b8acb282473336d0bc"
format_stamp: "Formatted at 2026-06-21 02:43:14 on dist-test-slave-jkn7"
server_key: "4580ba97cc4ceba1123f7baf4ba9d47a"
server_key_iv: "465e60e2bc872bd97591d91d193512fd"
server_key_version: "encryptionkey@0"
I20260621 02:43:14.423121 20289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260621 02:43:14.426450 20303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:14.427716 20289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260621 02:43:14.427901 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/wal
uuid: "1cca920b7a6644b8acb282473336d0bc"
format_stamp: "Formatted at 2026-06-21 02:43:14 on dist-test-slave-jkn7"
server_key: "4580ba97cc4ceba1123f7baf4ba9d47a"
server_key_iv: "465e60e2bc872bd97591d91d193512fd"
server_key_version: "encryptionkey@0"
I20260621 02:43:14.428118 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:14.446934 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:14.447683 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:14.447959 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:14.448880 20289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:14.450362 20289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:14.450480 20289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:14.450613 20289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:14.450706 20289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:14.484139 20289 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42279
I20260621 02:43:14.484221 20415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42279 every 8 connection(s)
I20260621 02:43:14.485468 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/data/info.pb
I20260621 02:43:14.487102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 20289
I20260621 02:43:14.487270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-0/wal/instance
I20260621 02:43:14.487802 388 external_mini_cluster.cc:1428] Setting key 6faa90bde666c18b381551856183fe50
I20260621 02:43:14.490984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--builtin_ntp_servers=127.0.97.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:43:14.500799 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:14.488152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50867 (local address 127.0.97.62:41885)
0621 02:43:14.488602 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:14.488608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:14.490072 (+ 1464us) server_negotiation.cc:407] Connection header received
0621 02:43:14.492842 (+ 2770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:14.492846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:14.492917 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:14.493010 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:14.494406 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:14.495119 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:14.497615 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:14.498103 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:14.498201 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:14.499806 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:14.499814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:14.499821 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:14.499847 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:14.499899 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:14.499918 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:14.500010 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:14.500286 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 02:43:14.500408 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":233,"threads_started":1}
I20260621 02:43:14.501642 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:14.488833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41885 (local address 127.0.97.1:50867)
0621 02:43:14.489900 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:14.489950 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:14.492561 (+ 2611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:14.493254 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:14.493268 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:14.493422 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:14.494177 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:14.494202 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:14.495273 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:14.495282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:14.497464 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:14.497482 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:14.498333 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:14.498343 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:14.498658 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:14.499408 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:43:14.499413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:14.499466 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:43:14.499570 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:14.500038 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:14.500048 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:14.500054 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:43:14.500218 (+ 164us) client_negotiation.cc:240] Negotiation successful
0621 02:43:14.500504 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":293,"threads_started":1}
I20260621 02:43:14.503217 20416 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41885
I20260621 02:43:14.503530 20416 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:14.504254 20416 heartbeater.cc:511] Master 127.0.97.62:41885 requested a full tablet report, sending...
I20260621 02:43:14.506470 20230 ts_manager.cc:194] Registered new tserver with Master: 1cca920b7a6644b8acb282473336d0bc (127.0.97.1:42279)
I20260621 02:43:14.507884 20230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50867
W20260621 02:43:14.818552 20420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:14.818885 20420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:14.819034 20420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:14.826012 20420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:14.826136 20420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:14.826305 20420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260621 02:43:14.835321 20420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:14.836539 20420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:14.838294 20420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:14.851069 20426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:14.851150 20428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:14.851097 20425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:14.851609 20420 server_base.cc:1034] running on GCE node
I20260621 02:43:14.852301 20420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:14.853219 20420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:14.854496 20420 hybrid_clock.cc:648] HybridClock initialized: now 1782009794854445 us; error 39 us; skew 500 ppm
I20260621 02:43:14.854842 20420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:14.856992 20420 webserver.cc:460] Webserver started at http://127.0.97.2:44223/ using document root <none> and password file <none>
I20260621 02:43:14.857731 20420 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:14.857863 20420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:14.858176 20420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:14.860913 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/data/instance:
uuid: "ca27becbe3a9495d82f2066fe7fb9f9a"
format_stamp: "Formatted at 2026-06-21 02:43:14 on dist-test-slave-jkn7"
server_key: "668aacac1476e7da548130ba25a8a690"
server_key_iv: "998ce3fdaea3e537ddd086cc41e4fd20"
server_key_version: "encryptionkey@0"
I20260621 02:43:14.861721 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ca27becbe3a9495d82f2066fe7fb9f9a"
format_stamp: "Formatted at 2026-06-21 02:43:14 on dist-test-slave-jkn7"
server_key: "668aacac1476e7da548130ba25a8a690"
server_key_iv: "998ce3fdaea3e537ddd086cc41e4fd20"
server_key_version: "encryptionkey@0"
I20260621 02:43:14.866384 20420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:43:14.869589 20434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:14.870808 20420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:43:14.870934 20420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/wal
uuid: "ca27becbe3a9495d82f2066fe7fb9f9a"
format_stamp: "Formatted at 2026-06-21 02:43:14 on dist-test-slave-jkn7"
server_key: "668aacac1476e7da548130ba25a8a690"
server_key_iv: "998ce3fdaea3e537ddd086cc41e4fd20"
server_key_version: "encryptionkey@0"
I20260621 02:43:14.871101 20420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:14.896101 20420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:14.896844 20420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:14.897073 20420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:14.897967 20420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:14.899333 20420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:14.899428 20420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:14.899506 20420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:14.899565 20420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:14.933483 20420 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45443
I20260621 02:43:14.933531 20546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45443 every 8 connection(s)
I20260621 02:43:14.934774 20420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/data/info.pb
I20260621 02:43:14.938349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 20420
I20260621 02:43:14.938534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/wal/instance
I20260621 02:43:14.939081 388 external_mini_cluster.cc:1428] Setting key 4ca086863e5ccdf07eab1a900f828cba
I20260621 02:43:14.941816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--builtin_ntp_servers=127.0.97.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:43:14.947172 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:14.937361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55187 (local address 127.0.97.62:41885)
0621 02:43:14.937574 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:14.937579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:14.939137 (+ 1558us) server_negotiation.cc:407] Connection header received
0621 02:43:14.940911 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:14.940916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:14.941006 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:14.941115 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:14.942246 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:14.942951 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:14.944470 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:14.944868 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:14.945005 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:14.946387 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:14.946392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:14.946395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:14.946411 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:14.946443 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:14.946454 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:14.946555 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:14.946771 (+ 216us) server_negotiation.cc:299] Negotiation successful
0621 02:43:14.946870 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:43:14.948238 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:14.937866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41885 (local address 127.0.97.2:55187)
0621 02:43:14.938985 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:14.939027 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:14.940651 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:14.941389 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:14.941397 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:14.941485 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:14.942092 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:14.942111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:14.943110 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:14.943115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:14.944322 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:14.944334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:14.945089 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:14.945096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:14.945374 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:14.946073 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:43:14.946076 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:14.946112 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:43:14.946187 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:14.946553 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:14.946564 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:14.946574 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:43:14.946746 (+ 172us) client_negotiation.cc:240] Negotiation successful
0621 02:43:14.947122 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":333,"threads_started":1}
I20260621 02:43:14.949662 20547 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41885
I20260621 02:43:14.949971 20547 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:14.950811 20547 heartbeater.cc:511] Master 127.0.97.62:41885 requested a full tablet report, sending...
I20260621 02:43:14.952294 20230 ts_manager.cc:194] Registered new tserver with Master: ca27becbe3a9495d82f2066fe7fb9f9a (127.0.97.2:45443)
I20260621 02:43:14.953087 20230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:55187
W20260621 02:43:15.276455 20550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:15.276712 20550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:15.276875 20550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:15.283955 20550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:15.284047 20550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:15.284214 20550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260621 02:43:15.293025 20550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:15.294353 20550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:15.296089 20550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:15.309146 20555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:15.309098 20556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:15.309170 20558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:15.309764 20550 server_base.cc:1034] running on GCE node
I20260621 02:43:15.310514 20550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:15.311492 20550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:15.312752 20550 hybrid_clock.cc:648] HybridClock initialized: now 1782009795312686 us; error 31 us; skew 500 ppm
I20260621 02:43:15.313097 20550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:15.315349 20550 webserver.cc:460] Webserver started at http://127.0.97.3:41947/ using document root <none> and password file <none>
I20260621 02:43:15.316092 20550 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:15.316224 20550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:15.316632 20550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:15.319681 20550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/data/instance:
uuid: "50cc5caf46874d3cb5b03b408c959f8e"
format_stamp: "Formatted at 2026-06-21 02:43:15 on dist-test-slave-jkn7"
server_key: "24d7eadec3aee201f09585574cb3808e"
server_key_iv: "e78a8e3d75dc268167b32999968facba"
server_key_version: "encryptionkey@0"
I20260621 02:43:15.320466 20550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/wal/instance:
uuid: "50cc5caf46874d3cb5b03b408c959f8e"
format_stamp: "Formatted at 2026-06-21 02:43:15 on dist-test-slave-jkn7"
server_key: "24d7eadec3aee201f09585574cb3808e"
server_key_iv: "e78a8e3d75dc268167b32999968facba"
server_key_version: "encryptionkey@0"
I20260621 02:43:15.325224 20550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260621 02:43:15.328836 20564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:15.330192 20550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:43:15.330366 20550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/wal
uuid: "50cc5caf46874d3cb5b03b408c959f8e"
format_stamp: "Formatted at 2026-06-21 02:43:15 on dist-test-slave-jkn7"
server_key: "24d7eadec3aee201f09585574cb3808e"
server_key_iv: "e78a8e3d75dc268167b32999968facba"
server_key_version: "encryptionkey@0"
I20260621 02:43:15.330585 20550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:15.385360 20550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:15.386281 20550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:15.386575 20550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:15.387612 20550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:43:15.389181 20550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:43:15.389323 20550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:15.389477 20550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:43:15.389567 20550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:15.423237 20550 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34809
I20260621 02:43:15.423292 20676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34809 every 8 connection(s)
I20260621 02:43:15.424487 20550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/data/info.pb
I20260621 02:43:15.427732 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 20550
I20260621 02:43:15.427918 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/wal/instance
I20260621 02:43:15.428475 388 external_mini_cluster.cc:1428] Setting key 0efdc0f4e984c82bdabfaf7d6699aaa4
I20260621 02:43:15.436715 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.426991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60449 (local address 127.0.97.62:41885)
0621 02:43:15.427176 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.427180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.428642 (+ 1462us) server_negotiation.cc:407] Connection header received
0621 02:43:15.430452 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.430459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.430563 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.430684 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:15.431701 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.432496 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.434122 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.434514 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.434648 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.435932 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:15.435937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:15.435941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:15.435956 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:15.435986 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:15.435997 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:15.436092 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.436301 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.436398 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:43:15.437742 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.427408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41885 (local address 127.0.97.3:60449)
0621 02:43:15.428460 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:15.428509 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:15.430191 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:15.430914 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:15.430922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:15.431003 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:15.431531 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.431548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.432649 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.432653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.433978 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.433990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.434726 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.434736 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.434978 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.435613 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:43:15.435615 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:15.435651 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:43:15.435727 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:15.436090 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:15.436102 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:15.436112 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:43:15.436287 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 02:43:15.436693 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":337,"threads_started":1}
I20260621 02:43:15.438995 20677 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41885
I20260621 02:43:15.439288 20677 heartbeater.cc:464] Registering TS with master...
I20260621 02:43:15.440006 20677 heartbeater.cc:511] Master 127.0.97.62:41885 requested a full tablet report, sending...
I20260621 02:43:15.441352 20230 ts_manager.cc:194] Registered new tserver with Master: 50cc5caf46874d3cb5b03b408c959f8e (127.0.97.3:34809)
I20260621 02:43:15.442194 20230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:60449
I20260621 02:43:15.443588 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:43:15.458276 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.448169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.0.97.62:41885)
0621 02:43:15.448422 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.448429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.448656 (+ 227us) server_negotiation.cc:407] Connection header received
0621 02:43:15.448819 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.448824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.448896 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.448998 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:15.449503 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.450177 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.451557 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.451973 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.452081 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.457454 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:15.457460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:15.457463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:15.457486 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:15.457525 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:15.457538 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:15.457636 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.457814 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.457905 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260621 02:43:15.462769 20230 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:43:15.469817 20230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:43:15.494539 20689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.482635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45443 (local address 127.0.0.1:38484)
0621 02:43:15.483488 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:15.483539 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:15.483730 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:15.484881 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:15.484892 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:15.484936 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:15.485420 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.485438 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.487977 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.487985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.491069 (+ 3084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.491091 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.493821 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.493828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.494006 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.494043 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:43:15.494153 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 02:43:15.494292 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":397,"threads_started":1}
I20260621 02:43:15.495004 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.482789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.0.97.2:45443)
0621 02:43:15.483537 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.483546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.483757 (+ 211us) server_negotiation.cc:407] Connection header received
0621 02:43:15.483954 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.483965 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.484290 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.484523 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:15.485655 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.487759 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.491371 (+ 3612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.493408 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.493579 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.493922 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.494246 (+ 324us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.494515 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":460,"threads_started":1}
I20260621 02:43:15.500017 20482 tablet_service.cc:1467] Processing CreateTablet for tablet ffeb9e898bcf415883432186e59c0c98 (DEFAULT_TABLE table=test-table [id=91671f6320b447569ce7673a1cc4509a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:15.501622 20482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffeb9e898bcf415883432186e59c0c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:15.502895 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.484270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.0.97.3:34809)
0621 02:43:15.493787 (+ 9517us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.493796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.494965 (+ 1169us) server_negotiation.cc:407] Connection header received
0621 02:43:15.495164 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.495173 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.495448 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.495602 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:15.496411 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.498186 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.500329 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.501642 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.501787 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.502074 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.502357 (+ 283us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.502517 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9331,"thread_start_us":9179,"threads_started":1}
I20260621 02:43:15.503634 20689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.483892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34809 (local address 127.0.0.1:58932)
0621 02:43:15.494820 (+ 10928us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:15.494839 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:15.494961 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:15.495945 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:15.495950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:15.495976 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:15.496180 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.496194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.498321 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.498326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.500121 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.500156 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.501897 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.501904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.502129 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.502202 (+ 73us) client_negotiation.cc:769] Sending connection context
0621 02:43:15.502312 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 02:43:15.502494 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10822,"thread_start_us":11404,"threads_started":1}
I20260621 02:43:15.505806 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.495686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42279 (local address 127.0.0.1:44632)
0621 02:43:15.497831 (+ 2145us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:15.497849 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:15.497968 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:15.498551 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:15.498556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:15.498581 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:15.499008 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.499019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.500859 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.500864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.502944 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.502957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.505052 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.505061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.505254 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.505311 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 02:43:15.505406 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:43:15.505530 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1996}
I20260621 02:43:15.506299 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.496012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.0.97.1:42279)
0621 02:43:15.497792 (+ 1780us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.497801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.497959 (+ 158us) server_negotiation.cc:407] Connection header received
0621 02:43:15.498117 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.498125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.498356 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.498541 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:15.499198 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.500677 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.503165 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.504534 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.504664 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.504936 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.505474 (+ 538us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.505637 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1589,"thread_start_us":1479,"threads_started":1}
I20260621 02:43:15.507586 20612 tablet_service.cc:1467] Processing CreateTablet for tablet ffeb9e898bcf415883432186e59c0c98 (DEFAULT_TABLE table=test-table [id=91671f6320b447569ce7673a1cc4509a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:15.509091 20612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffeb9e898bcf415883432186e59c0c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:15.509796 20351 tablet_service.cc:1467] Processing CreateTablet for tablet ffeb9e898bcf415883432186e59c0c98 (DEFAULT_TABLE table=test-table [id=91671f6320b447569ce7673a1cc4509a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:43:15.510831 20416 heartbeater.cc:503] Master 127.0.97.62:41885 was elected leader, sending a full tablet report...
I20260621 02:43:15.511178 20351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffeb9e898bcf415883432186e59c0c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:15.516649 20694 tablet_bootstrap.cc:492] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: Bootstrap starting.
I20260621 02:43:15.520351 20694 tablet_bootstrap.cc:654] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:15.521607 20694 log.cc:824] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:15.522346 20695 tablet_bootstrap.cc:492] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: Bootstrap starting.
I20260621 02:43:15.524421 20697 tablet_bootstrap.cc:492] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc: Bootstrap starting.
I20260621 02:43:15.526171 20695 tablet_bootstrap.cc:654] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:15.527395 20695 log.cc:824] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:15.527810 20697 tablet_bootstrap.cc:654] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:15.527926 20694 tablet_bootstrap.cc:492] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: No bootstrap required, opened a new log
I20260621 02:43:15.528211 20694 ts_tablet_manager.cc:1397] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260621 02:43:15.528982 20697 log.cc:824] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:15.530624 20695 tablet_bootstrap.cc:492] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: No bootstrap required, opened a new log
I20260621 02:43:15.530890 20695 ts_tablet_manager.cc:1397] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 02:43:15.531845 20697 tablet_bootstrap.cc:492] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc: No bootstrap required, opened a new log
I20260621 02:43:15.532100 20697 ts_tablet_manager.cc:1397] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:43:15.538805 20695 raft_consensus.cc:348] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } }
I20260621 02:43:15.538828 20694 raft_consensus.cc:348] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } }
I20260621 02:43:15.539188 20695 raft_consensus.cc:374] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:15.539265 20694 raft_consensus.cc:374] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:15.539294 20695 raft_consensus.cc:729] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cc5caf46874d3cb5b03b408c959f8e, State: Initialized, Role: FOLLOWER
I20260621 02:43:15.539366 20694 raft_consensus.cc:729] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca27becbe3a9495d82f2066fe7fb9f9a, State: Initialized, Role: FOLLOWER
I20260621 02:43:15.539924 20695 consensus_queue.cc:260] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } }
I20260621 02:43:15.540050 20694 consensus_queue.cc:260] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } }
I20260621 02:43:15.541330 20547 heartbeater.cc:503] Master 127.0.97.62:41885 was elected leader, sending a full tablet report...
I20260621 02:43:15.541918 20694 ts_tablet_manager.cc:1428] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260621 02:43:15.541970 20677 heartbeater.cc:503] Master 127.0.97.62:41885 was elected leader, sending a full tablet report...
I20260621 02:43:15.542448 20697 raft_consensus.cc:348] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } }
I20260621 02:43:15.542827 20697 raft_consensus.cc:374] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:15.542927 20697 raft_consensus.cc:729] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cca920b7a6644b8acb282473336d0bc, State: Initialized, Role: FOLLOWER
I20260621 02:43:15.543715 20695 ts_tablet_manager.cc:1428] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260621 02:43:15.543648 20697 consensus_queue.cc:260] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } }
I20260621 02:43:15.545311 20697 ts_tablet_manager.cc:1428] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260621 02:43:15.638036 20702 raft_consensus.cc:482] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:43:15.638386 20702 raft_consensus.cc:504] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } }
I20260621 02:43:15.640056 20702 leader_election.cc:288] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca27becbe3a9495d82f2066fe7fb9f9a (127.0.97.2:45443), 50cc5caf46874d3cb5b03b408c959f8e (127.0.97.3:34809)
I20260621 02:43:15.646581 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.640539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41799 (local address 127.0.97.2:45443)
0621 02:43:15.640718 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.640722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.641253 (+ 531us) server_negotiation.cc:407] Connection header received
0621 02:43:15.641511 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.641517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.641614 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.641744 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:15.642405 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.643087 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.644801 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.645689 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.645830 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.645921 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.646167 (+ 246us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.646292 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:43:15.646602 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.640448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45443 (local address 127.0.97.1:41799)
0621 02:43:15.641134 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:15.641156 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:15.641341 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:15.641761 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:15.641773 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:15.641896 (+ 123us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:15.642260 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.642269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.643212 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.643219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.644613 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.644625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.645860 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.645866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.646014 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.646038 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 02:43:15.646142 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 02:43:15.646266 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":378,"threads_started":1}
I20260621 02:43:15.647440 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.641573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41997 (local address 127.0.97.3:34809)
0621 02:43:15.641799 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.641809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.642311 (+ 502us) server_negotiation.cc:407] Connection header received
0621 02:43:15.642508 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.642512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.642570 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.642666 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:43:15.643158 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.643843 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.645569 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.646401 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.646526 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.646619 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.647092 (+ 473us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.647222 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:43:15.647456 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.641269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34809 (local address 127.0.97.1:41997)
0621 02:43:15.642191 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:15.642213 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:15.642361 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:15.642679 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:15.642683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:15.642705 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:43:15.642996 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.643004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.643984 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.643989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.645381 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:15.645394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.646777 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.646783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:15.646931 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.646953 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:43:15.647056 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:43:15.647172 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":591,"threads_started":1}
I20260621 02:43:15.647747 20502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffeb9e898bcf415883432186e59c0c98" candidate_uuid: "1cca920b7a6644b8acb282473336d0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" is_pre_election: true
I20260621 02:43:15.648259 20502 raft_consensus.cc:2454] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cca920b7a6644b8acb282473336d0bc in term 0.
I20260621 02:43:15.648623 20632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffeb9e898bcf415883432186e59c0c98" candidate_uuid: "1cca920b7a6644b8acb282473336d0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50cc5caf46874d3cb5b03b408c959f8e" is_pre_election: true
I20260621 02:43:15.649024 20306 leader_election.cc:302] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cca920b7a6644b8acb282473336d0bc, ca27becbe3a9495d82f2066fe7fb9f9a; no voters:
I20260621 02:43:15.649165 20632 raft_consensus.cc:2454] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cca920b7a6644b8acb282473336d0bc in term 0.
I20260621 02:43:15.649505 20702 raft_consensus.cc:2781] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:43:15.649660 20702 raft_consensus.cc:482] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:43:15.649761 20702 raft_consensus.cc:3037] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:15.652119 20702 raft_consensus.cc:504] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } }
I20260621 02:43:15.652626 20702 leader_election.cc:288] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [CANDIDATE]: Term 1 election: Requested vote from peers ca27becbe3a9495d82f2066fe7fb9f9a (127.0.97.2:45443), 50cc5caf46874d3cb5b03b408c959f8e (127.0.97.3:34809)
I20260621 02:43:15.653141 20502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffeb9e898bcf415883432186e59c0c98" candidate_uuid: "1cca920b7a6644b8acb282473336d0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a"
I20260621 02:43:15.653159 20632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffeb9e898bcf415883432186e59c0c98" candidate_uuid: "1cca920b7a6644b8acb282473336d0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50cc5caf46874d3cb5b03b408c959f8e"
I20260621 02:43:15.653384 20632 raft_consensus.cc:3037] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:15.653385 20502 raft_consensus.cc:3037] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:15.656554 20632 raft_consensus.cc:2454] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cca920b7a6644b8acb282473336d0bc in term 1.
I20260621 02:43:15.656558 20502 raft_consensus.cc:2454] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cca920b7a6644b8acb282473336d0bc in term 1.
I20260621 02:43:15.656900 20306 leader_election.cc:302] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cca920b7a6644b8acb282473336d0bc, ca27becbe3a9495d82f2066fe7fb9f9a; no voters:
I20260621 02:43:15.657151 20702 raft_consensus.cc:2781] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:15.657590 20702 raft_consensus.cc:686] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 1 LEADER]: Becoming Leader. State: Replica: 1cca920b7a6644b8acb282473336d0bc, State: Running, Role: LEADER
I20260621 02:43:15.658036 20702 consensus_queue.cc:237] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } }
I20260621 02:43:15.662637 20228 catalog_manager.cc:5495] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cca920b7a6644b8acb282473336d0bc (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1cca920b7a6644b8acb282473336d0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cca920b7a6644b8acb282473336d0bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42279 } health_report { overall_health: HEALTHY } } }
W20260621 02:43:15.676187 20678 tablet.cc:2369] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:43:15.686885 20548 tablet.cc:2369] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:43:15.699410 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.694040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.0.97.1:42279)
0621 02:43:15.694243 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.694247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.694349 (+ 102us) server_negotiation.cc:407] Connection header received
0621 02:43:15.694533 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.694537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.694646 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.694793 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:15.695106 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.695791 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.697195 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.697644 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.697749 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.698304 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:15.698862 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:15.698970 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.699069 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.699151 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:43:15.708003 20632 raft_consensus.cc:1261] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [term 1 FOLLOWER]: Refusing update from remote peer 1cca920b7a6644b8acb282473336d0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:15.708003 20502 raft_consensus.cc:1261] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [term 1 FOLLOWER]: Refusing update from remote peer 1cca920b7a6644b8acb282473336d0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:15.708990 20705 consensus_queue.cc:1035] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "50cc5caf46874d3cb5b03b408c959f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:15.709394 20702 consensus_queue.cc:1035] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca27becbe3a9495d82f2066fe7fb9f9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:15.720158 20708 mvcc.cc:205] Tried to move back new op lower bound from 7299112123209461760 to 7299112123020431360. Current Snapshot: MvccSnapshot[applied={T|T < 7299112123209461760}]
I20260621 02:43:15.726962 20710 mvcc.cc:205] Tried to move back new op lower bound from 7299112123209461760 to 7299112123020431360. Current Snapshot: MvccSnapshot[applied={T|T < 7299112123209461760}]
I20260621 02:43:15.734807 20230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:15.735146 20230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:43:15.738505 20230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3f92ba3542a410e8d4cbfaf7c7fdd3a: Sending DeleteTablet for 3 replicas of tablet ffeb9e898bcf415883432186e59c0c98
I20260621 02:43:15.739630 20482 tablet_service.cc:1514] Processing DeleteTablet for tablet ffeb9e898bcf415883432186e59c0c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:15 UTC) from {username='slave'} at 127.0.0.1:38484
I20260621 02:43:15.739820 20351 tablet_service.cc:1514] Processing DeleteTablet for tablet ffeb9e898bcf415883432186e59c0c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:15 UTC) from {username='slave'} at 127.0.0.1:44632
I20260621 02:43:15.740336 20612 tablet_service.cc:1514] Processing DeleteTablet for tablet ffeb9e898bcf415883432186e59c0c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:43:15 UTC) from {username='slave'} at 127.0.0.1:58932
I20260621 02:43:15.742221 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20289
I20260621 02:43:15.752658 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.743545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.0.97.1:42279)
0621 02:43:15.743719 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.743726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.744170 (+ 444us) server_negotiation.cc:407] Connection header received
0621 02:43:15.744444 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.744451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.744572 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.744648 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:15.745522 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.746748 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.748674 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.749386 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.749482 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.750928 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:15.750939 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:15.750954 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:15.751753 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:15.751808 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:15.751833 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:15.751917 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.752160 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.752278 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:43:15.758136 20411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:15.762530 20719 tablet_replica.cc:330] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: stopping tablet replica
I20260621 02:43:15.763091 20717 tablet_replica.cc:330] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: stopping tablet replica
I20260621 02:43:15.763170 20719 raft_consensus.cc:2229] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:15.763600 20717 raft_consensus.cc:2229] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:43:15.763942 20719 raft_consensus.cc:2258] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:15.764194 20717 raft_consensus.cc:2258] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:15.766501 20717 ts_tablet_manager.cc:1905] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:15.766501 20719 ts_tablet_manager.cc:1905] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:15.774029 20717 ts_tablet_manager.cc:1918] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:15.774284 20717 log.cc:1196] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-1/wal/wals/ffeb9e898bcf415883432186e59c0c98
I20260621 02:43:15.774924 20719 ts_tablet_manager.cc:1918] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:43:15.775090 20719 log.cc:1196] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: Deleting WAL directory at /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009483336832-388-0/minicluster-data/ts-2/wal/wals/ffeb9e898bcf415883432186e59c0c98
I20260621 02:43:15.775205 20717 ts_tablet_manager.cc:1939] T ffeb9e898bcf415883432186e59c0c98 P ca27becbe3a9495d82f2066fe7fb9f9a: Deleting consensus metadata
I20260621 02:43:15.775753 20719 ts_tablet_manager.cc:1939] T ffeb9e898bcf415883432186e59c0c98 P 50cc5caf46874d3cb5b03b408c959f8e: Deleting consensus metadata
I20260621 02:43:15.777637 20216 catalog_manager.cc:4841] TS 50cc5caf46874d3cb5b03b408c959f8e (127.0.97.3:34809): tablet ffeb9e898bcf415883432186e59c0c98 (table test-table [id=91671f6320b447569ce7673a1cc4509a]) successfully deleted
I20260621 02:43:15.777981 20216 catalog_manager.cc:4841] TS ca27becbe3a9495d82f2066fe7fb9f9a (127.0.97.2:45443): tablet ffeb9e898bcf415883432186e59c0c98 (table test-table [id=91671f6320b447569ce7673a1cc4509a]) successfully deleted
I20260621 02:43:15.871493 20718 tablet_replica.cc:330] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc: stopping tablet replica
I20260621 02:43:15.872087 20718 raft_consensus.cc:2229] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:43:15.872829 20718 raft_consensus.cc:2258] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:43:15.875000 20718 ts_tablet_manager.cc:1905] T ffeb9e898bcf415883432186e59c0c98 P 1cca920b7a6644b8acb282473336d0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:43:15.877571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 20289
I20260621 02:43:15.895988 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20420
W20260621 02:43:15.896175 20216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42279 (error 108)
I20260621 02:43:15.898032 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.897601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42279 (local address 127.0.0.1:44668)
0621 02:43:15.897796 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:15.897893 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260621 02:43:15.898252 20216 catalog_manager.cc:4584] TS 1cca920b7a6644b8acb282473336d0bc (127.0.97.1:42279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffeb9e898bcf415883432186e59c0c98: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42279: connect: Connection refused (error 111)
I20260621 02:43:15.906324 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.896776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.0.97.2:45443)
0621 02:43:15.896965 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:15.896970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:15.897019 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:43:15.897227 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:15.897231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:15.897380 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:15.897495 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:15.897922 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.898826 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.900026 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:15.900441 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:15.900575 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:15.904842 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:15.904857 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:15.904871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:15.905540 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:15.905590 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:15.905611 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:15.905735 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:15.905868 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 02:43:15.905990 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:43:15.907016 20542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:15.951982 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.951503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42279 (local address 127.0.0.1:44674)
0621 02:43:15.951716 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:15.951817 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260621 02:43:15.952979 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:15.952653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42279 (local address 127.0.0.1:44684)
0621 02:43:15.952814 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:15.952877 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 02:43:16.005909 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 20420
I20260621 02:43:16.013494 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:16.012961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42279 (local address 127.0.0.1:44700)
0621 02:43:16.013185 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:16.013274 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 02:43:16.014580 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:16.014221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42279 (local address 127.0.0.1:44712)
0621 02:43:16.014397 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:16.014475 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:43:16.023787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20550
I20260621 02:43:16.030452 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:16.024559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.0.97.3:34809)
0621 02:43:16.024767 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:16.024772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:16.024790 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:43:16.024844 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:16.024848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:16.024945 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:16.025030 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:16.025442 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:16.026187 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:16.027396 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:16.027784 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:16.027876 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:16.029008 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:16.029017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:16.029031 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:16.029695 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:16.029742 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:16.029766 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:16.029849 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:16.030024 (+ 175us) server_negotiation.cc:299] Negotiation successful
0621 02:43:16.030152 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:43:16.031040 20672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:16.095211 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:16.094634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42279 (local address 127.0.0.1:44714)
0621 02:43:16.094895 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:16.095007 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260621 02:43:16.096483 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:16.096021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42279 (local address 127.0.0.1:44728)
0621 02:43:16.096243 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:16.096339 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260621 02:43:16.118304 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 20550
I20260621 02:43:16.134559 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20199
I20260621 02:43:16.135594 20260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:16.242863 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 20199
2026-06-21T02:43:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2769 ms)
[----------] 6 tests from EncryptionPolicyTest (21523 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:16 dist-test-slave-jkn7 krb5kdc[20728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:16 dist-test-slave-jkn7 krb5kdc[20728](info): set up 2 sockets
Jun 21 02:43:16 dist-test-slave-jkn7 krb5kdc[20728](info): commencing operation
krb5kdc: starting...
W20260621 02:43:17.341718 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:17 dist-test-slave-jkn7 krb5kdc[20728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:17.582590 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:17.907889 20741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:17.908170 20741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:17.908334 20741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:17.915481 20741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:17.915587 20741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:17.915678 20741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:17.915763 20741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:17.915875 20741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:17.925628 20741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:17.926901 20741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:17.928191 20741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:17.939760 20749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:17.939841 20747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:17.939808 20746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:17.940881 20741 server_base.cc:1034] running on GCE node
I20260621 02:43:17.941675 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:17.943020 20741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:17.944219 20741 hybrid_clock.cc:648] HybridClock initialized: now 1782009797944187 us; error 28 us; skew 500 ppm
I20260621 02:43:17.944603 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:17 dist-test-slave-jkn7 krb5kdc[20728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:17.947613 20741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:17.948977 20741 webserver.cc:460] Webserver started at http://127.0.0.1:39279/ using document root <none> and password file <none>
I20260621 02:43:17.949693 20741 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:17.949839 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:17.950130 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:17.952936 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "162b8f0ab52f479bbc8270dc1396110a"
format_stamp: "Formatted at 2026-06-21 02:43:17 on dist-test-slave-jkn7"
server_key: "192b5366d3fdd41a6e477a0e3ff8cd0b"
server_key_iv: "7df2aa08e955b825317639332f8cb232"
server_key_version: "encryptionkey@0"
I20260621 02:43:17.953694 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "162b8f0ab52f479bbc8270dc1396110a"
format_stamp: "Formatted at 2026-06-21 02:43:17 on dist-test-slave-jkn7"
server_key: "192b5366d3fdd41a6e477a0e3ff8cd0b"
server_key_iv: "7df2aa08e955b825317639332f8cb232"
server_key_version: "encryptionkey@0"
I20260621 02:43:17.958508 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260621 02:43:17.961838 20756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:17.963060 20741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:43:17.963261 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "162b8f0ab52f479bbc8270dc1396110a"
format_stamp: "Formatted at 2026-06-21 02:43:17 on dist-test-slave-jkn7"
server_key: "192b5366d3fdd41a6e477a0e3ff8cd0b"
server_key_iv: "7df2aa08e955b825317639332f8cb232"
server_key_version: "encryptionkey@0"
I20260621 02:43:17.963451 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:17.981192 20741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:17.985152 20741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:17.985466 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:18.003805 20741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34087
I20260621 02:43:18.003841 20807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34087 every 8 connection(s)
I20260621 02:43:18.005213 20741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:18.005751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 20741
I20260621 02:43:18.006037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:18.006932 388 external_mini_cluster.cc:1428] Setting key 3301794cf9d7fe30446d502415d2e721
I20260621 02:43:18.010177 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:43:18 dist-test-slave-jkn7 krb5kdc[20728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:43:18.024633 20808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a: Bootstrap starting.
I20260621 02:43:18.028076 20808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:18.029165 20808 log.cc:824] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:18.031199 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:18.008949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40489 (local address 127.0.0.1:34087)
0621 02:43:18.009753 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:18.009765 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:18.009807 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:43:18.011512 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:18.011535 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:18.011915 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:18.012240 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:18.014352 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:18.015689 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:18.017778 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:18.018486 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:18.018631 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:18.023818 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:18.023826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:18.023841 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:18.023877 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:18.027008 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:18.027774 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:18.027781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:18.027787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:18.027948 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:18.028489 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:18.028493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:18.028495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:18.028983 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:18.029326 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:18.029861 (+ 535us) server_negotiation.cc:299] Negotiation successful
0621 02:43:18.030221 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":396,"threads_started":1}
I20260621 02:43:18.032261 20808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a: No bootstrap required, opened a new log
I20260621 02:43:18.035722 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:18.044706 20808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162b8f0ab52f479bbc8270dc1396110a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } }
I20260621 02:43:18.045348 20808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:18.045538 20808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162b8f0ab52f479bbc8270dc1396110a, State: Initialized, Role: FOLLOWER
I20260621 02:43:18.046321 20808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162b8f0ab52f479bbc8270dc1396110a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } }
I20260621 02:43:18.046665 20808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:18.046826 20808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:18.047030 20808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:18.050602 20808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162b8f0ab52f479bbc8270dc1396110a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } }
I20260621 02:43:18.051319 20808 leader_election.cc:302] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 162b8f0ab52f479bbc8270dc1396110a; no voters:
I20260621 02:43:18.052078 20808 leader_election.cc:288] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:18.052150 20818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:18.054316 20818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [term 1 LEADER]: Becoming Leader. State: Replica: 162b8f0ab52f479bbc8270dc1396110a, State: Running, Role: LEADER
I20260621 02:43:18.055002 20818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162b8f0ab52f479bbc8270dc1396110a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } }
I20260621 02:43:18.055243 20808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:18.063529 20819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "162b8f0ab52f479bbc8270dc1396110a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162b8f0ab52f479bbc8270dc1396110a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } } }
I20260621 02:43:18.064046 20819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:18.066100 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:18.041866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54951 (local address 127.0.0.1:34087)
0621 02:43:18.042087 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:18.042093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:18.042722 (+ 629us) server_negotiation.cc:407] Connection header received
0621 02:43:18.042942 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:18.042947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:18.043016 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:18.043102 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:18.045367 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:18.046327 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:18.048415 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:18.048901 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:18.049010 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:18.052473 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:18.052483 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:18.052491 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:18.052560 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:18.057991 (+ 5431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:18.059390 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:18.059399 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:18.059405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:18.059597 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:18.060273 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:18.060280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:18.060287 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:18.060773 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:18.061170 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:18.065195 (+ 4025us) server_negotiation.cc:299] Negotiation successful
0621 02:43:18.065509 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:43:18.070032 20820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 162b8f0ab52f479bbc8270dc1396110a. Latest consensus state: current_term: 1 leader_uuid: "162b8f0ab52f479bbc8270dc1396110a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162b8f0ab52f479bbc8270dc1396110a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } } }
I20260621 02:43:18.070391 20820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:18.070966 20827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:18.076396 20827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:18.085470 20827 catalog_manager.cc:1347] Generated new cluster ID: bea36e86a347402aafcfde9e83ce7afd
I20260621 02:43:18.085600 20827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:18.096462 20827 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:18.098249 20827 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:18.109788 20827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 162b8f0ab52f479bbc8270dc1396110a: Generated new TSK 0
I20260621 02:43:18.110924 20827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:18.128541 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20741
I20260621 02:43:18.130103 20803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:18.221947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 20741
W20260621 02:43:18.238860 20726 connection.cc:535] client connection to 127.0.0.1:34087 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34087): Connection reset by peer (error 104)
2026-06-21T02:43:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:18 dist-test-slave-jkn7 krb5kdc[20840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:18 dist-test-slave-jkn7 krb5kdc[20840](info): set up 2 sockets
Jun 21 02:43:18 dist-test-slave-jkn7 krb5kdc[20840](info): commencing operation
krb5kdc: starting...
W20260621 02:43:19.333554 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:19 dist-test-slave-jkn7 krb5kdc[20840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:19.568658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:19.884356 20853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:19.884685 20853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:19.884853 20853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:19.892333 20853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:19.892489 20853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:19.892608 20853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:19.892717 20853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:19.892800 20853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:19.902621 20853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:19.903937 20853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:19.905259 20853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:19.916787 20858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:19.916761 20859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:19.916841 20861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:19.917830 20853 server_base.cc:1034] running on GCE node
I20260621 02:43:19.918637 20853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:19.920015 20853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:19.921273 20853 hybrid_clock.cc:648] HybridClock initialized: now 1782009799921222 us; error 54 us; skew 500 ppm
I20260621 02:43:19.921736 20853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:19 dist-test-slave-jkn7 krb5kdc[20840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:19.924875 20853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:19.926448 20853 webserver.cc:460] Webserver started at http://127.0.0.1:44293/ using document root <none> and password file <none>
I20260621 02:43:19.927143 20853 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:19.927289 20853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:19.927605 20853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:19.930620 20853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "da5a901a708948309197701b1e8c10ce"
format_stamp: "Formatted at 2026-06-21 02:43:19 on dist-test-slave-jkn7"
server_key: "624bd0de6fa779db0be4296f84501e77"
server_key_iv: "4de699c69b75dede3b2704ded9d3784d"
server_key_version: "encryptionkey@0"
I20260621 02:43:19.931392 20853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "da5a901a708948309197701b1e8c10ce"
format_stamp: "Formatted at 2026-06-21 02:43:19 on dist-test-slave-jkn7"
server_key: "624bd0de6fa779db0be4296f84501e77"
server_key_iv: "4de699c69b75dede3b2704ded9d3784d"
server_key_version: "encryptionkey@0"
I20260621 02:43:19.936563 20853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:43:19.940151 20868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:19.941552 20853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:43:19.941736 20853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "da5a901a708948309197701b1e8c10ce"
format_stamp: "Formatted at 2026-06-21 02:43:19 on dist-test-slave-jkn7"
server_key: "624bd0de6fa779db0be4296f84501e77"
server_key_iv: "4de699c69b75dede3b2704ded9d3784d"
server_key_version: "encryptionkey@0"
I20260621 02:43:19.941937 20853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:19.963994 20853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:19.968304 20853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:19.968726 20853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:19.987828 20853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36259
I20260621 02:43:19.987881 20919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36259 every 8 connection(s)
I20260621 02:43:19.989280 20853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:19.989696 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 20853
I20260621 02:43:19.989893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:19.990550 388 external_mini_cluster.cc:1428] Setting key 4861faf4458d53f121ce0345ae7a345d
I20260621 02:43:19.994163 20920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:20.003805 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:19.992397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39599 (local address 127.0.0.1:36259)
0621 02:43:19.993131 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:19.993147 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:19.993190 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:43:19.995008 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:19.995031 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:19.995409 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:19.995793 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:19.996329 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:19.997763 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:19.999121 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:19.999835 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:20.000044 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:20.001411 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:20.001419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:20.001435 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:20.002006 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:20.002091 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:20.002120 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:20.002303 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:20.002582 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 02:43:20.002777 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
I20260621 02:43:20.005455 20920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce: Bootstrap starting.
I20260621 02:43:20.007714 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:20.008849 20920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:20.009913 20920 log.cc:824] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:20.013072 20920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce: No bootstrap required, opened a new log
I20260621 02:43:20.019011 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:20.012169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47601 (local address 127.0.0.1:36259)
0621 02:43:20.012364 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:20.012370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:20.012685 (+ 315us) server_negotiation.cc:407] Connection header received
0621 02:43:20.012862 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:20.012866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:20.012923 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:20.013002 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:20.013631 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:20.014388 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:20.016319 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:20.016782 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:20.016877 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:20.018184 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:20.018189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:20.018193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:20.018213 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:20.018254 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:20.018269 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:20.018364 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:20.018582 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 02:43:20.018671 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:43:20.022221 20920 raft_consensus.cc:348] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5a901a708948309197701b1e8c10ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } }
I20260621 02:43:20.022629 20920 raft_consensus.cc:374] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:20.022730 20920 raft_consensus.cc:729] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5a901a708948309197701b1e8c10ce, State: Initialized, Role: FOLLOWER
I20260621 02:43:20.023392 20920 consensus_queue.cc:260] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5a901a708948309197701b1e8c10ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } }
I20260621 02:43:20.023669 20920 raft_consensus.cc:388] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:20.023818 20920 raft_consensus.cc:482] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:20.023981 20920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:20.026232 20920 raft_consensus.cc:504] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5a901a708948309197701b1e8c10ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } }
I20260621 02:43:20.026707 20920 leader_election.cc:302] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da5a901a708948309197701b1e8c10ce; no voters:
I20260621 02:43:20.027261 20920 leader_election.cc:288] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:20.027417 20930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:20.028091 20930 raft_consensus.cc:686] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [term 1 LEADER]: Becoming Leader. State: Replica: da5a901a708948309197701b1e8c10ce, State: Running, Role: LEADER
I20260621 02:43:20.028699 20930 consensus_queue.cc:237] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5a901a708948309197701b1e8c10ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } }
I20260621 02:43:20.029665 20920 sys_catalog.cc:564] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:20.031949 20931 sys_catalog.cc:455] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da5a901a708948309197701b1e8c10ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5a901a708948309197701b1e8c10ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } } }
I20260621 02:43:20.032454 20931 sys_catalog.cc:458] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:20.031945 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader da5a901a708948309197701b1e8c10ce. Latest consensus state: current_term: 1 leader_uuid: "da5a901a708948309197701b1e8c10ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5a901a708948309197701b1e8c10ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } } }
I20260621 02:43:20.032752 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:20.034409 20939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:20.040167 20939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:20.050056 20939 catalog_manager.cc:1347] Generated new cluster ID: 3d487269f7a84b558af84b2488b4e8d1
I20260621 02:43:20.050199 20939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:20.063068 20939 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:20.064348 20939 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:20.074478 20939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da5a901a708948309197701b1e8c10ce: Generated new TSK 0
I20260621 02:43:20.075331 20939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:20.081216 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20853
W20260621 02:43:20.082265 20915 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39599
W20260621 02:43:20.083039 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20853: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:20.083241 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 20853
W20260621 02:43:20.100945 20838 connection.cc:535] client connection to 127.0.0.1:36259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36259): Connection reset by peer (error 104)
2026-06-21T02:43:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1859 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:20 dist-test-slave-jkn7 krb5kdc[20951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:20 dist-test-slave-jkn7 krb5kdc[20951](info): set up 2 sockets
Jun 21 02:43:20 dist-test-slave-jkn7 krb5kdc[20951](info): commencing operation
krb5kdc: starting...
W20260621 02:43:21.181622 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:21 dist-test-slave-jkn7 krb5kdc[20951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:21.434507 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:21.750526 20964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:21.750794 20964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:21.750882 20964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:21.757808 20964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:21.757900 20964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:21.757946 20964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:21.758008 20964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:21.758045 20964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:21.767585 20964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:21.768639 20964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:21.769887 20964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:21.781404 20970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:21.782054 20972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:21.782297 20969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:21.782948 20964 server_base.cc:1034] running on GCE node
I20260621 02:43:21.783941 20964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:21.785391 20964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:21.786625 20964 hybrid_clock.cc:648] HybridClock initialized: now 1782009801786582 us; error 37 us; skew 500 ppm
I20260621 02:43:21.786981 20964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:21 dist-test-slave-jkn7 krb5kdc[20951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:21.794898 20964 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:21.796381 20964 webserver.cc:460] Webserver started at http://127.0.0.1:39797/ using document root <none> and password file <none>
I20260621 02:43:21.797029 20964 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:21.797128 20964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:21.797459 20964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:21.800359 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "c5c3e5668e22423ab9e297d47385badd"
format_stamp: "Formatted at 2026-06-21 02:43:21 on dist-test-slave-jkn7"
server_key: "f6b0326254c279efeace7e20453ac1c4"
server_key_iv: "0d7baf488fdbf95fd767a298d9055464"
server_key_version: "encryptionkey@0"
I20260621 02:43:21.801080 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5c3e5668e22423ab9e297d47385badd"
format_stamp: "Formatted at 2026-06-21 02:43:21 on dist-test-slave-jkn7"
server_key: "f6b0326254c279efeace7e20453ac1c4"
server_key_iv: "0d7baf488fdbf95fd767a298d9055464"
server_key_version: "encryptionkey@0"
I20260621 02:43:21.806267 20964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:43:21.809646 20979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:21.810976 20964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:43:21.811163 20964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "c5c3e5668e22423ab9e297d47385badd"
format_stamp: "Formatted at 2026-06-21 02:43:21 on dist-test-slave-jkn7"
server_key: "f6b0326254c279efeace7e20453ac1c4"
server_key_iv: "0d7baf488fdbf95fd767a298d9055464"
server_key_version: "encryptionkey@0"
I20260621 02:43:21.811363 20964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:21.816942 20964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:21.821842 20964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:21.822211 20964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:21.840706 20964 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36583
I20260621 02:43:21.840754 21030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36583 every 8 connection(s)
I20260621 02:43:21.842092 20964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:21.845770 21031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:21.846874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 20964
I20260621 02:43:21.847067 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:21.847656 388 external_mini_cluster.cc:1428] Setting key dc9a18487ee853c5c0e4540a6f10ebee
I20260621 02:43:21.855554 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:21.849732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54217 (local address 127.0.0.1:36583)
0621 02:43:21.850530 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:21.850542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:21.850589 (+ 47us) server_negotiation.cc:407] Connection header received
0621 02:43:21.852232 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:21.852258 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:21.852605 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:21.852948 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:21.854037 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:21.854043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:21.854055 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:21.854068 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:21.854123 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:21.854145 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:21.854305 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:21.854445 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 02:43:21.854656 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":345,"threads_started":1}
I20260621 02:43:21.859135 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd: Bootstrap starting.
I20260621 02:43:21.859193 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:21.862958 21031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:21.864310 21031 log.cc:824] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:21.866910 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:21.864467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38603 (local address 127.0.0.1:36583)
0621 02:43:21.864681 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:21.864687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:21.865011 (+ 324us) server_negotiation.cc:407] Connection header received
0621 02:43:21.865185 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:21.865190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:21.865257 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:21.865390 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:21.866257 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:21.866261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:21.866264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:21.866278 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:21.866316 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:21.866330 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:21.866418 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:21.866548 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 02:43:21.866594 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 02:43:21.868610 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd: No bootstrap required, opened a new log
I20260621 02:43:21.876168 21031 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3e5668e22423ab9e297d47385badd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260621 02:43:21.876560 21031 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:21.876637 21031 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c3e5668e22423ab9e297d47385badd, State: Initialized, Role: FOLLOWER
I20260621 02:43:21.877251 21031 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3e5668e22423ab9e297d47385badd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260621 02:43:21.877537 21031 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:21.877650 21031 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:21.877779 21031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:21.880115 21031 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3e5668e22423ab9e297d47385badd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260621 02:43:21.880774 21031 leader_election.cc:302] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5c3e5668e22423ab9e297d47385badd; no voters:
I20260621 02:43:21.881461 21031 leader_election.cc:288] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:21.881609 21041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:21.882292 21041 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [term 1 LEADER]: Becoming Leader. State: Replica: c5c3e5668e22423ab9e297d47385badd, State: Running, Role: LEADER
I20260621 02:43:21.882850 21041 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3e5668e22423ab9e297d47385badd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260621 02:43:21.883692 21031 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:21.885655 21042 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5c3e5668e22423ab9e297d47385badd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3e5668e22423ab9e297d47385badd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } } }
I20260621 02:43:21.886097 21042 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:21.886428 21043 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5c3e5668e22423ab9e297d47385badd. Latest consensus state: current_term: 1 leader_uuid: "c5c3e5668e22423ab9e297d47385badd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3e5668e22423ab9e297d47385badd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } } }
I20260621 02:43:21.886857 21043 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:21.888916 21046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:21.895215 21046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:21.904829 21046 catalog_manager.cc:1347] Generated new cluster ID: 31e7852fbaaf4970952e04b8690ca25e
I20260621 02:43:21.904960 21046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:21.923347 21046 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:21.923887 21046 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:21.940836 21046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5c3e5668e22423ab9e297d47385badd: Generated new TSK 0
I20260621 02:43:21.941936 21046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:21.955736 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20964
W20260621 02:43:21.956820 21026 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54217
W20260621 02:43:21.957579 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 20964: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:21.957738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 20964
2026-06-21T02:43:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:22 dist-test-slave-jkn7 krb5kdc[21062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:22 dist-test-slave-jkn7 krb5kdc[21062](info): set up 2 sockets
Jun 21 02:43:22 dist-test-slave-jkn7 krb5kdc[21062](info): commencing operation
krb5kdc: starting...
W20260621 02:43:23.061506 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:23 dist-test-slave-jkn7 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:23.308451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:23.639460 21075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:23.639712 21075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:23.639873 21075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:23.646744 21075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:23.646840 21075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:23.646973 21075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:23.647087 21075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:23.647172 21075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:23.656636 21075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:23.657958 21075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:23.659255 21075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:23.670401 21081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:23.670822 21080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:23.670622 21083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:23.671643 21075 server_base.cc:1034] running on GCE node
I20260621 02:43:23.672425 21075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:23.673786 21075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:23.675005 21075 hybrid_clock.cc:648] HybridClock initialized: now 1782009803674956 us; error 45 us; skew 500 ppm
I20260621 02:43:23.675331 21075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:23 dist-test-slave-jkn7 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:23.678390 21075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:23.679924 21075 webserver.cc:460] Webserver started at http://127.0.0.1:33193/ using document root <none> and password file <none>
I20260621 02:43:23.680606 21075 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:23.680755 21075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:23.681054 21075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:23.683946 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "b702be4c9c164c458bfb8c5248ea3b33"
format_stamp: "Formatted at 2026-06-21 02:43:23 on dist-test-slave-jkn7"
server_key: "4fe79106e065c88cd6480da0d41b2150"
server_key_iv: "6fec16e253dd74789aded3b8920b7779"
server_key_version: "encryptionkey@0"
I20260621 02:43:23.684693 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "b702be4c9c164c458bfb8c5248ea3b33"
format_stamp: "Formatted at 2026-06-21 02:43:23 on dist-test-slave-jkn7"
server_key: "4fe79106e065c88cd6480da0d41b2150"
server_key_iv: "6fec16e253dd74789aded3b8920b7779"
server_key_version: "encryptionkey@0"
I20260621 02:43:23.689689 21075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:43:23.693189 21090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:23.694497 21075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:43:23.694717 21075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "b702be4c9c164c458bfb8c5248ea3b33"
format_stamp: "Formatted at 2026-06-21 02:43:23 on dist-test-slave-jkn7"
server_key: "4fe79106e065c88cd6480da0d41b2150"
server_key_iv: "6fec16e253dd74789aded3b8920b7779"
server_key_version: "encryptionkey@0"
I20260621 02:43:23.694932 21075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:23.706441 21075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:23.710330 21075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:23.710610 21075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:23.728739 21075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41033
I20260621 02:43:23.728783 21141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41033 every 8 connection(s)
I20260621 02:43:23.730160 21075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:23.733898 21142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:23.740358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 21075
I20260621 02:43:23.740546 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:23.741110 388 external_mini_cluster.cc:1428] Setting key 65cdbb2cca4fe2a6fc62278afe310b7a
I20260621 02:43:23.744007 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33: Bootstrap starting.
I20260621 02:43:23.747493 21142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:23.748621 21142 log.cc:824] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:23.751629 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33: No bootstrap required, opened a new log
Jun 21 02:43:23 dist-test-slave-jkn7 krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:43:23.760972 21142 raft_consensus.cc:348] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702be4c9c164c458bfb8c5248ea3b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } }
I20260621 02:43:23.761425 21142 raft_consensus.cc:374] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:23.761544 21142 raft_consensus.cc:729] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b702be4c9c164c458bfb8c5248ea3b33, State: Initialized, Role: FOLLOWER
I20260621 02:43:23.762099 21142 consensus_queue.cc:260] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702be4c9c164c458bfb8c5248ea3b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } }
I20260621 02:43:23.762354 21142 raft_consensus.cc:388] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:23.762512 21142 raft_consensus.cc:482] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:23.762694 21142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:23.764437 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:23.742917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55425 (local address 127.0.0.1:41033)
0621 02:43:23.743676 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:23.743688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:23.743728 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:43:23.745504 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:23.745526 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:23.745902 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:23.746206 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:23.748096 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:23.749509 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:23.750926 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:23.751913 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:23.752139 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:23.756046 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:23.756055 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:23.756076 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:23.756129 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:23.760140 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:23.760806 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:23.760814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:23.760821 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:23.761036 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:23.761566 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:23.761573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:23.761576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:23.762241 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:23.762545 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:23.762981 (+ 436us) server_negotiation.cc:299] Negotiation successful
0621 02:43:23.763425 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":359,"threads_started":1}
I20260621 02:43:23.765158 21142 raft_consensus.cc:504] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702be4c9c164c458bfb8c5248ea3b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } }
I20260621 02:43:23.765748 21142 leader_election.cc:302] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b702be4c9c164c458bfb8c5248ea3b33; no voters:
I20260621 02:43:23.766366 21142 leader_election.cc:288] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:23.766475 21147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:23.767735 21147 raft_consensus.cc:686] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [term 1 LEADER]: Becoming Leader. State: Replica: b702be4c9c164c458bfb8c5248ea3b33, State: Running, Role: LEADER
I20260621 02:43:23.768445 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:23.768299 21147 consensus_queue.cc:237] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702be4c9c164c458bfb8c5248ea3b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } }
I20260621 02:43:23.768493 21142 sys_catalog.cc:564] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:23.770781 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b702be4c9c164c458bfb8c5248ea3b33. Latest consensus state: current_term: 1 leader_uuid: "b702be4c9c164c458bfb8c5248ea3b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702be4c9c164c458bfb8c5248ea3b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } } }
I20260621 02:43:23.771163 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:23.771715 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b702be4c9c164c458bfb8c5248ea3b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702be4c9c164c458bfb8c5248ea3b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41033 } } }
I20260621 02:43:23.772017 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:23.794222 21152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:23.808959 21152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:23.822433 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:23.792995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36773 (local address 127.0.0.1:41033)
0621 02:43:23.793228 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:23.793237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:23.800190 (+ 6953us) server_negotiation.cc:407] Connection header received
0621 02:43:23.800710 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:23.800719 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:23.800832 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:23.800936 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:23.803614 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:23.805101 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:23.807745 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:23.808717 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:23.808856 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:23.812954 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:23.812982 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:23.812990 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:23.813041 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:23.818201 (+ 5160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:23.819275 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:23.819283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:23.819289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:23.819467 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:23.820003 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:23.820011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:23.820016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:23.820532 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:23.820831 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:23.821571 (+ 740us) server_negotiation.cc:299] Negotiation successful
0621 02:43:23.821860 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:43:23.832252 21152 catalog_manager.cc:1347] Generated new cluster ID: 8a9d94528f1b4757915e9702a973fcc4
I20260621 02:43:23.832379 21152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:23.851831 21152 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:23.853664 21152 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:23.864249 21152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b702be4c9c164c458bfb8c5248ea3b33: Generated new TSK 0
I20260621 02:43:23.865223 21152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:23.881911 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21075
I20260621 02:43:23.882880 21137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:23.974606 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 21075
2026-06-21T02:43:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:24 dist-test-slave-jkn7 krb5kdc[21175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:24 dist-test-slave-jkn7 krb5kdc[21175](info): set up 2 sockets
Jun 21 02:43:24 dist-test-slave-jkn7 krb5kdc[21175](info): commencing operation
krb5kdc: starting...
W20260621 02:43:25.101609 388 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:25 dist-test-slave-jkn7 krb5kdc[21175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:25.336505 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:25.682832 21188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:25.683130 21188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:25.683310 21188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:25.690622 21188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:25.690728 21188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:25.690856 21188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:25.690961 21188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:25.691047 21188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:25.701531 21188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:25.702955 21188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:25.704299 21188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:25.716495 21193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:25.716424 21194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:25.716464 21196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:25.717355 21188 server_base.cc:1034] running on GCE node
I20260621 02:43:25.718147 21188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:25.719628 21188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:25.720854 21188 hybrid_clock.cc:648] HybridClock initialized: now 1782009805720794 us; error 47 us; skew 500 ppm
I20260621 02:43:25.721218 21188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:25 dist-test-slave-jkn7 krb5kdc[21175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:25.724347 21188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:25.725868 21188 webserver.cc:460] Webserver started at http://127.0.0.1:43707/ using document root <none> and password file <none>
I20260621 02:43:25.726543 21188 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:25.726687 21188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:25.726984 21188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:25.729924 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "28cc268dba7f4ec89e8418d26c51c0e0"
format_stamp: "Formatted at 2026-06-21 02:43:25 on dist-test-slave-jkn7"
server_key: "f2e88c90e99b9d7fd63b692ac62455bf"
server_key_iv: "73e1a5cd561f48877a98d7e6de476c5f"
server_key_version: "encryptionkey@0"
I20260621 02:43:25.730669 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "28cc268dba7f4ec89e8418d26c51c0e0"
format_stamp: "Formatted at 2026-06-21 02:43:25 on dist-test-slave-jkn7"
server_key: "f2e88c90e99b9d7fd63b692ac62455bf"
server_key_iv: "73e1a5cd561f48877a98d7e6de476c5f"
server_key_version: "encryptionkey@0"
I20260621 02:43:25.735682 21188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:43:25.739100 21203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:25.740442 21188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:43:25.740635 21188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "28cc268dba7f4ec89e8418d26c51c0e0"
format_stamp: "Formatted at 2026-06-21 02:43:25 on dist-test-slave-jkn7"
server_key: "f2e88c90e99b9d7fd63b692ac62455bf"
server_key_iv: "73e1a5cd561f48877a98d7e6de476c5f"
server_key_version: "encryptionkey@0"
I20260621 02:43:25.740833 21188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:25.762221 21188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:25.766348 21188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:25.766640 21188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:25.785096 21188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35617
I20260621 02:43:25.785127 21254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35617 every 8 connection(s)
I20260621 02:43:25.786510 21188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:25.790525 21255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:25.792305 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 21188
I20260621 02:43:25.792496 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:25.793041 388 external_mini_cluster.cc:1428] Setting key d8c2a6bac3b1b755fc114300ec0e7f95
I20260621 02:43:25.806340 21255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0: Bootstrap starting.
I20260621 02:43:25.808387 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:25.794905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38699 (local address 127.0.0.1:35617)
0621 02:43:25.795738 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:25.795754 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:25.795852 (+ 98us) server_negotiation.cc:407] Connection header received
0621 02:43:25.798380 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:25.798411 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:25.798982 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:25.799443 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:25.799939 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:25.801818 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:25.803168 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:25.804248 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:25.804420 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:25.805566 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:25.805577 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:25.805594 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:25.806316 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:25.806416 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:25.806451 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:25.806564 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:25.806801 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 02:43:25.807078 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":385,"threads_started":1}
I20260621 02:43:25.810328 21255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:25.811614 21255 log.cc:824] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:25.812173 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:25.814342 21255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0: No bootstrap required, opened a new log
I20260621 02:43:25.822906 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:25.816515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50695 (local address 127.0.0.1:35617)
0621 02:43:25.816704 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:25.816710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:25.816998 (+ 288us) server_negotiation.cc:407] Connection header received
0621 02:43:25.817151 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:25.817157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:25.817232 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:25.817364 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:25.817838 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:25.819006 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:25.820280 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:25.820984 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:25.821113 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:25.822158 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:25.822165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:25.822170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:25.822189 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:25.822237 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:25.822263 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:25.822361 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:25.822475 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 02:43:25.822570 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:43:25.823905 21255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc268dba7f4ec89e8418d26c51c0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } }
I20260621 02:43:25.824245 21255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:25.824348 21255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cc268dba7f4ec89e8418d26c51c0e0, State: Initialized, Role: FOLLOWER
I20260621 02:43:25.824904 21255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc268dba7f4ec89e8418d26c51c0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } }
I20260621 02:43:25.825138 21255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:25.825286 21255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:25.825510 21255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:25.827662 21255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc268dba7f4ec89e8418d26c51c0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } }
I20260621 02:43:25.828158 21255 leader_election.cc:302] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28cc268dba7f4ec89e8418d26c51c0e0; no voters:
I20260621 02:43:25.828699 21255 leader_election.cc:288] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:25.828847 21265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:25.829715 21265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [term 1 LEADER]: Becoming Leader. State: Replica: 28cc268dba7f4ec89e8418d26c51c0e0, State: Running, Role: LEADER
I20260621 02:43:25.830292 21265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc268dba7f4ec89e8418d26c51c0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } }
I20260621 02:43:25.830883 21255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:25.833390 21267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28cc268dba7f4ec89e8418d26c51c0e0. Latest consensus state: current_term: 1 leader_uuid: "28cc268dba7f4ec89e8418d26c51c0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc268dba7f4ec89e8418d26c51c0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } } }
I20260621 02:43:25.833388 21266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28cc268dba7f4ec89e8418d26c51c0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28cc268dba7f4ec89e8418d26c51c0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } } }
I20260621 02:43:25.833860 21267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:25.833892 21266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:25.835574 21273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:25.846174 21273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:25.853675 21273 catalog_manager.cc:1347] Generated new cluster ID: dac6f2179d514d9994abe415ac9b8fb0
I20260621 02:43:25.853782 21273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:25.865808 21273 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:25.867110 21273 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:25.876957 21273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28cc268dba7f4ec89e8418d26c51c0e0: Generated new TSK 0
I20260621 02:43:25.877753 21273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:25.888561 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21188
W20260621 02:43:25.889549 21250 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38699
W20260621 02:43:25.890163 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21188: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:25.890317 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 21188
2026-06-21T02:43:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:25 dist-test-slave-jkn7 krb5kdc[21286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:25 dist-test-slave-jkn7 krb5kdc[21286](info): set up 2 sockets
Jun 21 02:43:25 dist-test-slave-jkn7 krb5kdc[21286](info): commencing operation
krb5kdc: starting...
W20260621 02:43:26.977586 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:27 dist-test-slave-jkn7 krb5kdc[21286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:27.206128 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:27.534169 21299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:27.534468 21299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:27.534680 21299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:27.542029 21299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:27.542138 21299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:27.542255 21299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:27.542358 21299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:27.542449 21299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:27.552464 21299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:27.553864 21299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:27.555193 21299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:27.567299 21307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:27.567288 21305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:27.567842 21304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:27.568599 21299 server_base.cc:1034] running on GCE node
I20260621 02:43:27.569478 21299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:27.570960 21299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:27.572224 21299 hybrid_clock.cc:648] HybridClock initialized: now 1782009807572176 us; error 32 us; skew 500 ppm
I20260621 02:43:27.572636 21299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:27 dist-test-slave-jkn7 krb5kdc[21286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:27.579677 21299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:27.581569 21299 webserver.cc:460] Webserver started at http://127.0.0.1:40143/ using document root <none> and password file <none>
I20260621 02:43:27.582297 21299 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:27.582466 21299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:27.582790 21299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:27.585915 21299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7"
format_stamp: "Formatted at 2026-06-21 02:43:27 on dist-test-slave-jkn7"
server_key: "b8f262ce3cbba42daa53a36e2dafdc9f"
server_key_iv: "ffe8d06688411b66635d61a55def7c48"
server_key_version: "encryptionkey@0"
I20260621 02:43:27.586763 21299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7"
format_stamp: "Formatted at 2026-06-21 02:43:27 on dist-test-slave-jkn7"
server_key: "b8f262ce3cbba42daa53a36e2dafdc9f"
server_key_iv: "ffe8d06688411b66635d61a55def7c48"
server_key_version: "encryptionkey@0"
I20260621 02:43:27.591953 21299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260621 02:43:27.595398 21314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:27.596690 21299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:43:27.596870 21299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7"
format_stamp: "Formatted at 2026-06-21 02:43:27 on dist-test-slave-jkn7"
server_key: "b8f262ce3cbba42daa53a36e2dafdc9f"
server_key_iv: "ffe8d06688411b66635d61a55def7c48"
server_key_version: "encryptionkey@0"
I20260621 02:43:27.597077 21299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:27.602815 21299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:27.607007 21299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:27.607306 21299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:27.625078 21299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46743
I20260621 02:43:27.625113 21365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46743 every 8 connection(s)
I20260621 02:43:27.626538 21299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:27.627650 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 21299
I20260621 02:43:27.627859 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:27.628302 388 external_mini_cluster.cc:1428] Setting key 92d848e416918e07807989440785f6b5
I20260621 02:43:27.631083 21366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:27.637171 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:27.630158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59913 (local address 127.0.0.1:46743)
0621 02:43:27.630963 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:27.630979 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:27.631032 (+ 53us) server_negotiation.cc:407] Connection header received
0621 02:43:27.633355 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:27.633385 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:27.633816 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:27.634283 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:27.635252 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:27.635261 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:27.635277 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:27.635293 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:27.635387 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:27.635423 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:27.635646 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:27.635780 (+ 134us) server_negotiation.cc:299] Negotiation successful
0621 02:43:27.635975 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":326,"threads_started":1}
I20260621 02:43:27.641040 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:27.645344 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7: Bootstrap starting.
I20260621 02:43:27.648501 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:27.646270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59093 (local address 127.0.0.1:46743)
0621 02:43:27.646437 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:27.646442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:27.646744 (+ 302us) server_negotiation.cc:407] Connection header received
0621 02:43:27.646867 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:27.646877 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:27.646955 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:27.647077 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:27.647931 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:27.647936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:27.647940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:27.647956 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:27.648002 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:27.648025 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:27.648148 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:27.648194 (+ 46us) server_negotiation.cc:299] Negotiation successful
0621 02:43:27.648245 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:43:27.648955 21366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:27.650188 21366 log.cc:824] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:27.652731 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7: No bootstrap required, opened a new log
I20260621 02:43:27.660017 21366 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } }
I20260621 02:43:27.660458 21366 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:27.660573 21366 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fc62e18a8a4f37934b9b69bfdfeec7, State: Initialized, Role: FOLLOWER
I20260621 02:43:27.661098 21366 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } }
I20260621 02:43:27.661345 21366 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:27.661497 21366 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:27.661664 21366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:27.663687 21366 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } }
I20260621 02:43:27.664144 21366 leader_election.cc:302] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2fc62e18a8a4f37934b9b69bfdfeec7; no voters:
I20260621 02:43:27.664690 21366 leader_election.cc:288] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:27.664832 21376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:27.665524 21376 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [term 1 LEADER]: Becoming Leader. State: Replica: a2fc62e18a8a4f37934b9b69bfdfeec7, State: Running, Role: LEADER
I20260621 02:43:27.666042 21376 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } }
I20260621 02:43:27.666560 21366 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:27.669003 21377 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } } }
I20260621 02:43:27.668975 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2fc62e18a8a4f37934b9b69bfdfeec7. Latest consensus state: current_term: 1 leader_uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fc62e18a8a4f37934b9b69bfdfeec7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46743 } } }
I20260621 02:43:27.669761 21377 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:27.669811 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:27.670516 21382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:27.675311 21382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:27.683045 21382 catalog_manager.cc:1347] Generated new cluster ID: d31fda36fd5247cdaa188e837efcdb62
I20260621 02:43:27.683145 21382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:27.696830 21382 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:27.697340 21382 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:27.706328 21382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2fc62e18a8a4f37934b9b69bfdfeec7: Generated new TSK 0
I20260621 02:43:27.707127 21382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:27.713927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21299
W20260621 02:43:27.715055 21361 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59913
W20260621 02:43:27.715796 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21299: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:27.715952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 21299
2026-06-21T02:43:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1823 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:27 dist-test-slave-jkn7 krb5kdc[21397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:27 dist-test-slave-jkn7 krb5kdc[21397](info): set up 2 sockets
Jun 21 02:43:27 dist-test-slave-jkn7 krb5kdc[21397](info): commencing operation
krb5kdc: starting...
W20260621 02:43:28.805670 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:28 dist-test-slave-jkn7 krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:29.046913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:29.373205 21410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:29.373584 21410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:29.373747 21410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:29.380900 21410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:29.381036 21410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:29.381137 21410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:29.381243 21410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:29.381340 21410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:29.391026 21410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:29.392313 21410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:29.393694 21410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:29.405346 21418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:29.405433 21415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:29.405382 21416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:29.406415 21410 server_base.cc:1034] running on GCE node
I20260621 02:43:29.407234 21410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:29.408672 21410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:29.409893 21410 hybrid_clock.cc:648] HybridClock initialized: now 1782009809409840 us; error 41 us; skew 500 ppm
I20260621 02:43:29.410240 21410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:29 dist-test-slave-jkn7 krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:29.413184 21410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:29.414685 21410 webserver.cc:460] Webserver started at http://127.0.0.1:44875/ using document root <none> and password file <none>
I20260621 02:43:29.415349 21410 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:29.415493 21410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:29.415798 21410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:29.418730 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "f8adfe633a664528a456e9c57c859761"
format_stamp: "Formatted at 2026-06-21 02:43:29 on dist-test-slave-jkn7"
server_key: "41ff095a8514008f34e7af9b962d6438"
server_key_iv: "bde72075d6d9f7a6571224e00c680c07"
server_key_version: "encryptionkey@0"
I20260621 02:43:29.419476 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "f8adfe633a664528a456e9c57c859761"
format_stamp: "Formatted at 2026-06-21 02:43:29 on dist-test-slave-jkn7"
server_key: "41ff095a8514008f34e7af9b962d6438"
server_key_iv: "bde72075d6d9f7a6571224e00c680c07"
server_key_version: "encryptionkey@0"
I20260621 02:43:29.424413 21410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:43:29.427774 21425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:29.428988 21410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:43:29.429178 21410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "f8adfe633a664528a456e9c57c859761"
format_stamp: "Formatted at 2026-06-21 02:43:29 on dist-test-slave-jkn7"
server_key: "41ff095a8514008f34e7af9b962d6438"
server_key_iv: "bde72075d6d9f7a6571224e00c680c07"
server_key_version: "encryptionkey@0"
I20260621 02:43:29.429445 21410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:29.447021 21410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:29.450980 21410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:29.451279 21410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:29.470103 21410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35639
I20260621 02:43:29.470160 21476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35639 every 8 connection(s)
I20260621 02:43:29.471487 21410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:29.471895 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 21410
I20260621 02:43:29.472101 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:29.472564 388 external_mini_cluster.cc:1428] Setting key 6bd52370af3e2aa51ecd85b1bc074e12
I20260621 02:43:29.476123 21477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:43:29 dist-test-slave-jkn7 krb5kdc[21397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:43:29.486145 21477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761: Bootstrap starting.
I20260621 02:43:29.489037 21477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:29.490105 21477 log.cc:824] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:29.493336 21477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761: No bootstrap required, opened a new log
I20260621 02:43:29.494501 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:29.474485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:47091 (local address 127.0.0.1:35639)
0621 02:43:29.475247 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:29.475259 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:29.475298 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:43:29.476985 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:29.477007 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:29.477440 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:29.477805 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:29.479206 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:29.480508 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:29.481844 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:29.482460 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:29.482668 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:29.486738 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:29.486746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:29.486758 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:29.486799 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:29.489840 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:29.490758 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:29.490764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:29.490769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:29.490933 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:29.491488 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:29.491495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:29.491498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:29.492125 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:29.492468 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:29.492855 (+ 387us) server_negotiation.cc:299] Negotiation successful
0621 02:43:29.493168 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":340,"threads_started":1}
I20260621 02:43:29.498320 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:29.501706 21477 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8adfe633a664528a456e9c57c859761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35639 } }
I20260621 02:43:29.502055 21477 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:29.502156 21477 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8adfe633a664528a456e9c57c859761, State: Initialized, Role: FOLLOWER
I20260621 02:43:29.502823 21477 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8adfe633a664528a456e9c57c859761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35639 } }
I20260621 02:43:29.503082 21477 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:29.503238 21477 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:29.503434 21477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:29.505615 21477 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8adfe633a664528a456e9c57c859761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35639 } }
I20260621 02:43:29.506178 21477 leader_election.cc:302] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8adfe633a664528a456e9c57c859761; no voters:
I20260621 02:43:29.506809 21477 leader_election.cc:288] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:29.506870 21487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:29.508785 21487 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [term 1 LEADER]: Becoming Leader. State: Replica: f8adfe633a664528a456e9c57c859761, State: Running, Role: LEADER
I20260621 02:43:29.509415 21477 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:29.509500 21487 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8adfe633a664528a456e9c57c859761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35639 } }
I20260621 02:43:29.515803 21489 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8adfe633a664528a456e9c57c859761. Latest consensus state: current_term: 1 leader_uuid: "f8adfe633a664528a456e9c57c859761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8adfe633a664528a456e9c57c859761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35639 } } }
I20260621 02:43:29.516201 21489 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:29.516626 21488 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8adfe633a664528a456e9c57c859761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8adfe633a664528a456e9c57c859761" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35639 } } }
I20260621 02:43:29.516981 21488 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:29.522387 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:29.502489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:57891 (local address 127.0.0.1:35639)
0621 02:43:29.502676 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:29.502684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:29.503194 (+ 510us) server_negotiation.cc:407] Connection header received
0621 02:43:29.503394 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:29.503399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:29.503476 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:29.503578 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:29.504886 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:29.505752 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:29.507590 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:29.508243 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:29.508360 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:29.511826 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:29.511845 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:29.511853 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:29.511903 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:29.516913 (+ 5010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:29.518308 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:29.518317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:29.518322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:29.518524 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:29.519258 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:29.519266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:29.519272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:29.519751 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:29.521359 (+ 1608us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:29.521583 (+ 224us) server_negotiation.cc:299] Negotiation successful
0621 02:43:29.521818 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:43:29.526263 21498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:29.531473 21498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:29.540373 21498 catalog_manager.cc:1347] Generated new cluster ID: 714127e1139a4cc8868130aad2f15738
I20260621 02:43:29.540477 21498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:29.558548 21498 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:29.559787 21498 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:29.565961 21498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8adfe633a664528a456e9c57c859761: Generated new TSK 0
I20260621 02:43:29.566766 21498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:29.579214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21410
I20260621 02:43:29.580740 21472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:29.672309 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 21410
W20260621 02:43:29.689128 21395 connection.cc:535] client connection to 127.0.0.1:35639 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35639): Connection reset by peer (error 104)
2026-06-21T02:43:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:29 dist-test-slave-jkn7 krb5kdc[21510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:29 dist-test-slave-jkn7 krb5kdc[21510](info): set up 2 sockets
Jun 21 02:43:29 dist-test-slave-jkn7 krb5kdc[21510](info): commencing operation
krb5kdc: starting...
W20260621 02:43:30.773597 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:30 dist-test-slave-jkn7 krb5kdc[21510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:31.010895 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:31.331161 21523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:31.331475 21523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:31.331674 21523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:31.339052 21523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:31.339191 21523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:31.339309 21523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:31.339414 21523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:31.339493 21523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:31.349632 21523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:31.351029 21523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:31.352478 21523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:31.368419 21529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:31.368451 21528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:31.368391 21531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:31.369194 21523 server_base.cc:1034] running on GCE node
I20260621 02:43:31.370054 21523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:31.371513 21523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:31.372731 21523 hybrid_clock.cc:648] HybridClock initialized: now 1782009811372687 us; error 41 us; skew 500 ppm
I20260621 02:43:31.373080 21523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:31 dist-test-slave-jkn7 krb5kdc[21510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:31.376231 21523 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:31.377811 21523 webserver.cc:460] Webserver started at http://127.0.0.1:42149/ using document root <none> and password file <none>
I20260621 02:43:31.378515 21523 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:31.378669 21523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:31.378981 21523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:31.382095 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "02a5510430b04737a7d06aa6f2405a89"
format_stamp: "Formatted at 2026-06-21 02:43:31 on dist-test-slave-jkn7"
server_key: "f7bc1a2a985fbac69b5b10b04ec0306d"
server_key_iv: "95a9125617c4cc27b8ac8d8d3cbbdb8a"
server_key_version: "encryptionkey@0"
I20260621 02:43:31.382884 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "02a5510430b04737a7d06aa6f2405a89"
format_stamp: "Formatted at 2026-06-21 02:43:31 on dist-test-slave-jkn7"
server_key: "f7bc1a2a985fbac69b5b10b04ec0306d"
server_key_iv: "95a9125617c4cc27b8ac8d8d3cbbdb8a"
server_key_version: "encryptionkey@0"
I20260621 02:43:31.388267 21523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:43:31.391831 21538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:31.393211 21523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:43:31.393453 21523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "02a5510430b04737a7d06aa6f2405a89"
format_stamp: "Formatted at 2026-06-21 02:43:31 on dist-test-slave-jkn7"
server_key: "f7bc1a2a985fbac69b5b10b04ec0306d"
server_key_iv: "95a9125617c4cc27b8ac8d8d3cbbdb8a"
server_key_version: "encryptionkey@0"
I20260621 02:43:31.393648 21523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:31.408829 21523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:31.413326 21523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:31.413617 21523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:31.431171 21523 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41213
I20260621 02:43:31.431208 21589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41213 every 8 connection(s)
I20260621 02:43:31.432555 21523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:31.433179 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 21523
I20260621 02:43:31.433427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:31.433995 388 external_mini_cluster.cc:1428] Setting key dd963000b27590ecb1713a9a64ea1a47
I20260621 02:43:31.436941 21590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:31.450313 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:31.436065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:58485 (local address 127.0.0.1:41213)
0621 02:43:31.437013 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:31.437057 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:31.437451 (+ 394us) server_negotiation.cc:407] Connection header received
0621 02:43:31.439890 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:31.439918 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:31.440473 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:31.440923 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:31.441487 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:31.443598 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:31.445037 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:31.446078 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:31.446258 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:31.447481 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:31.447491 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:31.447512 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:31.448259 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:31.448359 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:31.448393 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:31.448553 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:31.448830 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 02:43:31.449005 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":408,"threads_started":1}
I20260621 02:43:31.451285 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89: Bootstrap starting.
I20260621 02:43:31.454509 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:31.455228 21590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:31.456472 21590 log.cc:824] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:31.460070 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89: No bootstrap required, opened a new log
I20260621 02:43:31.466815 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:31.459913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:33939 (local address 127.0.0.1:41213)
0621 02:43:31.460113 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:31.460122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:31.460566 (+ 444us) server_negotiation.cc:407] Connection header received
0621 02:43:31.460786 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:31.460792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:31.460878 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:31.461001 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:31.461499 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:31.462677 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:31.464011 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:31.464720 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:31.464856 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:31.465997 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:31.466005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:31.466010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:31.466036 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:31.466089 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:31.466111 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:31.466236 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:31.466409 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 02:43:31.466467 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:43:31.469871 21590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a5510430b04737a7d06aa6f2405a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260621 02:43:31.470216 21590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:31.470289 21590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a5510430b04737a7d06aa6f2405a89, State: Initialized, Role: FOLLOWER
I20260621 02:43:31.470798 21590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a5510430b04737a7d06aa6f2405a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260621 02:43:31.471071 21590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:31.471257 21590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:31.471444 21590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:31.473773 21590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a5510430b04737a7d06aa6f2405a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260621 02:43:31.474320 21590 leader_election.cc:302] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02a5510430b04737a7d06aa6f2405a89; no voters:
I20260621 02:43:31.474985 21590 leader_election.cc:288] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:31.475134 21600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:31.475854 21600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [term 1 LEADER]: Becoming Leader. State: Replica: 02a5510430b04737a7d06aa6f2405a89, State: Running, Role: LEADER
I20260621 02:43:31.476347 21600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a5510430b04737a7d06aa6f2405a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260621 02:43:31.477015 21590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:31.479362 21602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a5510430b04737a7d06aa6f2405a89. Latest consensus state: current_term: 1 leader_uuid: "02a5510430b04737a7d06aa6f2405a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a5510430b04737a7d06aa6f2405a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } } }
I20260621 02:43:31.479899 21602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:31.479354 21601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02a5510430b04737a7d06aa6f2405a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a5510430b04737a7d06aa6f2405a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } } }
I20260621 02:43:31.481024 21601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:31.481338 21607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:31.488660 21607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:31.497618 21607 catalog_manager.cc:1347] Generated new cluster ID: 9d7554666f0e484b8461b86f233c587c
I20260621 02:43:31.497751 21607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:31.529844 21607 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:31.531726 21607 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:31.540117 21607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02a5510430b04737a7d06aa6f2405a89: Generated new TSK 0
I20260621 02:43:31.541193 21607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:31.561745 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21523
W20260621 02:43:31.562813 21585 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.12:58485
W20260621 02:43:31.563515 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21523: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:31.563671 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 21523
W20260621 02:43:31.580251 21508 connection.cc:535] client connection to 127.0.0.1:41213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41213): Connection reset by peer (error 104)
2026-06-21T02:43:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1890 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:31 dist-test-slave-jkn7 krb5kdc[21621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:31 dist-test-slave-jkn7 krb5kdc[21621](info): set up 2 sockets
Jun 21 02:43:31 dist-test-slave-jkn7 krb5kdc[21621](info): commencing operation
krb5kdc: starting...
W20260621 02:43:32.650044 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.007s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:32 dist-test-slave-jkn7 krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:32.900408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:33.230079 21634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:33.230394 21634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:33.230540 21634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:33.237619 21634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:33.237754 21634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:33.237855 21634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:33.237962 21634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:33.238035 21634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:33.247470 21634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:33.248638 21634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:33.249895 21634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:33.261078 21640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:33.261122 21639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:33.261122 21642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:33.261852 21634 server_base.cc:1034] running on GCE node
I20260621 02:43:33.262588 21634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:33.263871 21634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:33.265115 21634 hybrid_clock.cc:648] HybridClock initialized: now 1782009813265071 us; error 48 us; skew 500 ppm
I20260621 02:43:33.265506 21634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:33 dist-test-slave-jkn7 krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:33.272867 21634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:33.274256 21634 webserver.cc:460] Webserver started at http://127.0.0.1:37953/ using document root <none> and password file <none>
I20260621 02:43:33.274896 21634 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:33.275053 21634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:33.275482 21634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:33.278527 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "e31a3ad7be6249c6823a41e46bd4d552"
format_stamp: "Formatted at 2026-06-21 02:43:33 on dist-test-slave-jkn7"
server_key: "1ad3dce86d4e70acb486efec9eec9416"
server_key_iv: "a68b34c0af00bec012c2608124c3e06b"
server_key_version: "encryptionkey@0"
I20260621 02:43:33.279256 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "e31a3ad7be6249c6823a41e46bd4d552"
format_stamp: "Formatted at 2026-06-21 02:43:33 on dist-test-slave-jkn7"
server_key: "1ad3dce86d4e70acb486efec9eec9416"
server_key_iv: "a68b34c0af00bec012c2608124c3e06b"
server_key_version: "encryptionkey@0"
I20260621 02:43:33.284128 21634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260621 02:43:33.287339 21649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:33.288668 21634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:43:33.288827 21634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "e31a3ad7be6249c6823a41e46bd4d552"
format_stamp: "Formatted at 2026-06-21 02:43:33 on dist-test-slave-jkn7"
server_key: "1ad3dce86d4e70acb486efec9eec9416"
server_key_iv: "a68b34c0af00bec012c2608124c3e06b"
server_key_version: "encryptionkey@0"
I20260621 02:43:33.288995 21634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:33.294366 21634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:33.298357 21634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:33.298641 21634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:33.315665 21634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40485
I20260621 02:43:33.315708 21700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40485 every 8 connection(s)
I20260621 02:43:33.317090 21634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:33.320585 21701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:33.323992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 21634
I20260621 02:43:33.324198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:33.324726 388 external_mini_cluster.cc:1428] Setting key 30f9f6c247645a869eacc5c6b4c6be3c
I20260621 02:43:33.332394 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:33.326490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:41037 (local address 127.0.0.1:40485)
0621 02:43:33.327254 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:33.327265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:33.327306 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:43:33.328953 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:33.328975 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:33.329396 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:33.329734 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:33.330823 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:33.330829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:33.330842 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:33.330862 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:33.330942 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:33.330963 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:33.331147 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:33.331248 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 02:43:33.331462 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":370,"threads_started":1}
I20260621 02:43:33.333395 21701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552: Bootstrap starting.
I20260621 02:43:33.336229 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:33.336922 21701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:33.338029 21701 log.cc:824] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:33.341464 21701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552: No bootstrap required, opened a new log
I20260621 02:43:33.343474 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:33.340791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:33499 (local address 127.0.0.1:40485)
0621 02:43:33.341002 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:33.341008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:33.341325 (+ 317us) server_negotiation.cc:407] Connection header received
0621 02:43:33.341530 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:33.341534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:33.341618 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:33.341714 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:33.342762 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:33.342766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:33.342769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:33.342783 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:33.342826 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:33.342846 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:33.342933 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:33.343096 (+ 163us) server_negotiation.cc:299] Negotiation successful
0621 02:43:33.343148 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 02:43:33.349570 21701 raft_consensus.cc:348] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31a3ad7be6249c6823a41e46bd4d552" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } }
I20260621 02:43:33.349992 21701 raft_consensus.cc:374] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:33.350097 21701 raft_consensus.cc:729] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31a3ad7be6249c6823a41e46bd4d552, State: Initialized, Role: FOLLOWER
I20260621 02:43:33.350705 21701 consensus_queue.cc:260] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31a3ad7be6249c6823a41e46bd4d552" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } }
I20260621 02:43:33.350953 21701 raft_consensus.cc:388] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:33.351094 21701 raft_consensus.cc:482] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:33.351274 21701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:33.353765 21701 raft_consensus.cc:504] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31a3ad7be6249c6823a41e46bd4d552" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } }
I20260621 02:43:33.354351 21701 leader_election.cc:302] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e31a3ad7be6249c6823a41e46bd4d552; no voters:
I20260621 02:43:33.356074 21701 leader_election.cc:288] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:33.356192 21711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:33.357511 21711 raft_consensus.cc:686] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [term 1 LEADER]: Becoming Leader. State: Replica: e31a3ad7be6249c6823a41e46bd4d552, State: Running, Role: LEADER
I20260621 02:43:33.358304 21711 consensus_queue.cc:237] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31a3ad7be6249c6823a41e46bd4d552" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } }
I20260621 02:43:33.359862 21701 sys_catalog.cc:564] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:33.363217 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e31a3ad7be6249c6823a41e46bd4d552. Latest consensus state: current_term: 1 leader_uuid: "e31a3ad7be6249c6823a41e46bd4d552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31a3ad7be6249c6823a41e46bd4d552" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } } }
I20260621 02:43:33.363817 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:33.365777 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e31a3ad7be6249c6823a41e46bd4d552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31a3ad7be6249c6823a41e46bd4d552" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } } }
I20260621 02:43:33.366220 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:33.370463 21717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:33.374641 21717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:33.383143 21717 catalog_manager.cc:1347] Generated new cluster ID: acb50d05e460411dadf86e3ec9baa2ac
I20260621 02:43:33.383392 21717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:33.406178 21717 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:33.406664 21717 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:33.423071 21717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e31a3ad7be6249c6823a41e46bd4d552: Generated new TSK 0
I20260621 02:43:33.423986 21717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:33.429569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21634
W20260621 02:43:33.430657 21696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.12:41037
W20260621 02:43:33.431391 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21634: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:33.431557 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 21634
2026-06-21T02:43:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:33 dist-test-slave-jkn7 krb5kdc[21732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:33 dist-test-slave-jkn7 krb5kdc[21732](info): set up 2 sockets
Jun 21 02:43:33 dist-test-slave-jkn7 krb5kdc[21732](info): commencing operation
krb5kdc: starting...
W20260621 02:43:34.525702 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:34 dist-test-slave-jkn7 krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:34.766004 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:35.079979 21745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:35.080229 21745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:35.080425 21745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:35.087492 21745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:35.087607 21745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:35.087708 21745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:35.087802 21745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:35.087880 21745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:35.097136 21745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:35.098316 21745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:35.099467 21745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:35.110641 21750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:35.110580 21751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:35.110651 21753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:35.111421 21745 server_base.cc:1034] running on GCE node
I20260621 02:43:35.112136 21745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:35.113472 21745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:35.114677 21745 hybrid_clock.cc:648] HybridClock initialized: now 1782009815114631 us; error 48 us; skew 500 ppm
I20260621 02:43:35.114986 21745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:35 dist-test-slave-jkn7 krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:35.118024 21745 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:35.119331 21745 webserver.cc:460] Webserver started at http://127.0.0.1:42929/ using document root <none> and password file <none>
I20260621 02:43:35.119951 21745 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:35.120076 21745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:35.120424 21745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:35.123195 21745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "26ed77f1d8064cc39fe924aeb4482427"
format_stamp: "Formatted at 2026-06-21 02:43:35 on dist-test-slave-jkn7"
server_key: "cecdacf35846455771a709685e6787d9"
server_key_iv: "01edd4892a9bd9e98b67a0e249f56f71"
server_key_version: "encryptionkey@0"
I20260621 02:43:35.123920 21745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "26ed77f1d8064cc39fe924aeb4482427"
format_stamp: "Formatted at 2026-06-21 02:43:35 on dist-test-slave-jkn7"
server_key: "cecdacf35846455771a709685e6787d9"
server_key_iv: "01edd4892a9bd9e98b67a0e249f56f71"
server_key_version: "encryptionkey@0"
I20260621 02:43:35.128659 21745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260621 02:43:35.131809 21760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:35.132989 21745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 02:43:35.133169 21745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "26ed77f1d8064cc39fe924aeb4482427"
format_stamp: "Formatted at 2026-06-21 02:43:35 on dist-test-slave-jkn7"
server_key: "cecdacf35846455771a709685e6787d9"
server_key_iv: "01edd4892a9bd9e98b67a0e249f56f71"
server_key_version: "encryptionkey@0"
I20260621 02:43:35.133402 21745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:35.161057 21745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:35.164836 21745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:35.165110 21745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:35.182809 21745 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44561
I20260621 02:43:35.182847 21811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44561 every 8 connection(s)
I20260621 02:43:35.184089 21745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:35.186774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 21745
I20260621 02:43:35.186955 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:35.187465 388 external_mini_cluster.cc:1428] Setting key e4e786d9726c6f7d5b8d2342744dadf3
I20260621 02:43:35.187927 21812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:35.202287 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427: Bootstrap starting.
Jun 21 02:43:35 dist-test-slave-jkn7 krb5kdc[21732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:43:35.205585 21812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:35.206673 21812 log.cc:824] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:35.209583 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427: No bootstrap required, opened a new log
I20260621 02:43:35.211247 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:35.189197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:35259 (local address 127.0.0.1:44561)
0621 02:43:35.189983 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:35.189993 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:35.190032 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:43:35.192360 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:35.192381 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:35.192775 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:35.193095 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:35.194884 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:35.196127 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:35.197993 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:35.198694 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:35.198847 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:35.203710 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:35.203719 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:35.203733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:35.203776 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:35.207017 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:35.207858 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:35.207871 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:35.207877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:35.208055 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:35.208546 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:35.208550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:35.208552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:35.209070 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:35.209431 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:35.209952 (+ 521us) server_negotiation.cc:299] Negotiation successful
0621 02:43:35.210273 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":332,"threads_started":1}
I20260621 02:43:35.214951 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:35.217736 21812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed77f1d8064cc39fe924aeb4482427" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260621 02:43:35.218119 21812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:35.218228 21812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ed77f1d8064cc39fe924aeb4482427, State: Initialized, Role: FOLLOWER
I20260621 02:43:35.218768 21812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed77f1d8064cc39fe924aeb4482427" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260621 02:43:35.219008 21812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:35.219159 21812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:35.219374 21812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:35.222689 21812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed77f1d8064cc39fe924aeb4482427" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260621 02:43:35.223178 21812 leader_election.cc:302] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26ed77f1d8064cc39fe924aeb4482427; no voters:
I20260621 02:43:35.223747 21812 leader_election.cc:288] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:35.223886 21822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:35.224953 21822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [term 1 LEADER]: Becoming Leader. State: Replica: 26ed77f1d8064cc39fe924aeb4482427, State: Running, Role: LEADER
I20260621 02:43:35.225615 21822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed77f1d8064cc39fe924aeb4482427" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260621 02:43:35.226008 21812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:35.228358 21823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26ed77f1d8064cc39fe924aeb4482427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed77f1d8064cc39fe924aeb4482427" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } } }
I20260621 02:43:35.228739 21823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:35.231396 21824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26ed77f1d8064cc39fe924aeb4482427. Latest consensus state: current_term: 1 leader_uuid: "26ed77f1d8064cc39fe924aeb4482427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed77f1d8064cc39fe924aeb4482427" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } } }
I20260621 02:43:35.231715 21824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:35.233355 21827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:35.238687 21827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:35.239439 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:35.219158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:59867 (local address 127.0.0.1:44561)
0621 02:43:35.219373 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:35.219382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:35.219905 (+ 523us) server_negotiation.cc:407] Connection header received
0621 02:43:35.220076 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:35.220079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:35.220142 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:35.220240 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:35.222093 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:35.222835 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:35.224757 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:35.225529 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:35.225668 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:35.229129 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:35.229140 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:35.229148 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:35.229198 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:35.234052 (+ 4854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:35.235039 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:35.235046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:35.235052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:35.235244 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:35.235997 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:35.236004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:35.236010 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:35.236580 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:35.236950 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:35.238649 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0621 02:43:35.238898 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:43:35.252187 21827 catalog_manager.cc:1347] Generated new cluster ID: fa34e09f40de4ce5a72c67dfb77f80df
I20260621 02:43:35.252295 21827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:35.285451 21827 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:35.286664 21827 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:35.301371 21827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26ed77f1d8064cc39fe924aeb4482427: Generated new TSK 0
I20260621 02:43:35.302148 21827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:35.312206 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21745
I20260621 02:43:35.313611 21807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:35.410264 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 21745
W20260621 02:43:35.427649 21730 connection.cc:535] client connection to 127.0.0.1:44561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44561): Connection reset by peer (error 104)
2026-06-21T02:43:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:35 dist-test-slave-jkn7 krb5kdc[21844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:35 dist-test-slave-jkn7 krb5kdc[21844](info): set up 2 sockets
Jun 21 02:43:35 dist-test-slave-jkn7 krb5kdc[21844](info): commencing operation
krb5kdc: starting...
W20260621 02:43:36.517666 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:36 dist-test-slave-jkn7 krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:36.769018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:37.091777 21857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:37.092063 21857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:37.092221 21857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:37.099197 21857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:37.099294 21857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:37.099393 21857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:37.099489 21857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:37.099577 21857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:37.109084 21857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:37.110288 21857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:37.111521 21857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:37.123039 21862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:37.123001 21863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:37.123065 21865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:37.123746 21857 server_base.cc:1034] running on GCE node
I20260621 02:43:37.124478 21857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:37.125931 21857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:37.127126 21857 hybrid_clock.cc:648] HybridClock initialized: now 1782009817127087 us; error 34 us; skew 500 ppm
I20260621 02:43:37.127449 21857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:37 dist-test-slave-jkn7 krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:37.130563 21857 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:37.131964 21857 webserver.cc:460] Webserver started at http://127.0.0.1:38361/ using document root <none> and password file <none>
I20260621 02:43:37.132578 21857 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:37.132678 21857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:37.132916 21857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:37.135725 21857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "eb09ea846fed44509ae1fa457a0a658d"
format_stamp: "Formatted at 2026-06-21 02:43:37 on dist-test-slave-jkn7"
server_key: "250cc4214aa1e660ffb99d7fa4a62771"
server_key_iv: "2073dd1450749ec257b69cc5be721b05"
server_key_version: "encryptionkey@0"
I20260621 02:43:37.136404 21857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "eb09ea846fed44509ae1fa457a0a658d"
format_stamp: "Formatted at 2026-06-21 02:43:37 on dist-test-slave-jkn7"
server_key: "250cc4214aa1e660ffb99d7fa4a62771"
server_key_iv: "2073dd1450749ec257b69cc5be721b05"
server_key_version: "encryptionkey@0"
I20260621 02:43:37.141145 21857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260621 02:43:37.144389 21872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:37.145685 21857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:43:37.145823 21857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "eb09ea846fed44509ae1fa457a0a658d"
format_stamp: "Formatted at 2026-06-21 02:43:37 on dist-test-slave-jkn7"
server_key: "250cc4214aa1e660ffb99d7fa4a62771"
server_key_iv: "2073dd1450749ec257b69cc5be721b05"
server_key_version: "encryptionkey@0"
I20260621 02:43:37.145974 21857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:37.171094 21857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:37.175238 21857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:37.175482 21857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:37.192795 21857 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40171
I20260621 02:43:37.192834 21923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40171 every 8 connection(s)
I20260621 02:43:37.194154 21857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:37.197576 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:37.202509 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 21857
I20260621 02:43:37.202693 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:37.203157 388 external_mini_cluster.cc:1428] Setting key 0f26ee0b608bcc4ad593b7558e8c0d5b
I20260621 02:43:37.207916 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d: Bootstrap starting.
I20260621 02:43:37.210842 21924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:37.211791 21924 log.cc:824] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:37.214820 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d: No bootstrap required, opened a new log
I20260621 02:43:37.215595 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:37.204850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:56573 (local address 127.0.0.1:40171)
0621 02:43:37.205769 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:37.205781 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:37.205821 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:43:37.207424 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:37.207446 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:37.207801 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:37.208166 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:37.208692 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:37.210030 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:37.211350 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:37.212034 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:37.212229 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:37.213505 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:37.213511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:37.213524 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:37.214014 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:37.214079 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:37.214103 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:37.214215 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:37.214494 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 02:43:37.214659 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":485,"threads_started":1}
I20260621 02:43:37.219041 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:37.222378 21924 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb09ea846fed44509ae1fa457a0a658d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } }
I20260621 02:43:37.222726 21924 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:37.222818 21924 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb09ea846fed44509ae1fa457a0a658d, State: Initialized, Role: FOLLOWER
I20260621 02:43:37.223516 21924 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb09ea846fed44509ae1fa457a0a658d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } }
I20260621 02:43:37.223830 21924 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:37.223992 21924 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:37.224191 21924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:37.227195 21924 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb09ea846fed44509ae1fa457a0a658d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } }
I20260621 02:43:37.227769 21924 leader_election.cc:302] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb09ea846fed44509ae1fa457a0a658d; no voters:
I20260621 02:43:37.228403 21924 leader_election.cc:288] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:37.228500 21934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:37.230672 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:37.223370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:41065 (local address 127.0.0.1:40171)
0621 02:43:37.223588 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:37.223594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:37.223951 (+ 357us) server_negotiation.cc:407] Connection header received
0621 02:43:37.224157 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:37.224162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:37.224229 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:37.224346 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:37.224998 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:37.225837 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:37.227645 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:37.228113 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:37.228281 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:37.229643 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:37.229649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:37.229654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:37.229676 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:37.229718 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:37.229739 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:37.229856 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:37.230283 (+ 427us) server_negotiation.cc:299] Negotiation successful
0621 02:43:37.230335 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:43:37.231043 21924 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:37.231328 21934 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [term 1 LEADER]: Becoming Leader. State: Replica: eb09ea846fed44509ae1fa457a0a658d, State: Running, Role: LEADER
I20260621 02:43:37.231931 21934 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb09ea846fed44509ae1fa457a0a658d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } }
I20260621 02:43:37.235396 21935 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb09ea846fed44509ae1fa457a0a658d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb09ea846fed44509ae1fa457a0a658d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } } }
I20260621 02:43:37.235776 21935 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:37.236248 21936 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb09ea846fed44509ae1fa457a0a658d. Latest consensus state: current_term: 1 leader_uuid: "eb09ea846fed44509ae1fa457a0a658d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb09ea846fed44509ae1fa457a0a658d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } } }
I20260621 02:43:37.236828 21936 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:37.237839 21943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:37.241995 21943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:37.249943 21943 catalog_manager.cc:1347] Generated new cluster ID: 2e60fac1e377424f8ec570cbc0ce5a00
I20260621 02:43:37.250054 21943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:37.268926 21943 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:37.270143 21943 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:37.280093 21943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb09ea846fed44509ae1fa457a0a658d: Generated new TSK 0
I20260621 02:43:37.281006 21943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:37.291268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21857
W20260621 02:43:37.292625 21919 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.12:56573
W20260621 02:43:37.293332 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21857: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:37.293481 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 21857
W20260621 02:43:37.310227 21842 connection.cc:535] client connection to 127.0.0.1:40171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40171): Connection reset by peer (error 104)
2026-06-21T02:43:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:37 dist-test-slave-jkn7 krb5kdc[21955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:37 dist-test-slave-jkn7 krb5kdc[21955](info): set up 2 sockets
Jun 21 02:43:37 dist-test-slave-jkn7 krb5kdc[21955](info): commencing operation
krb5kdc: starting...
W20260621 02:43:38.393537 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:38 dist-test-slave-jkn7 krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:43:38.631999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:38.961385 21968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:38.961722 21968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:38.961877 21968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:38.969000 21968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:38.969157 21968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:38.969264 21968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:38.969377 21968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:38.969451 21968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:38.979286 21968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:38.980577 21968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:38.981879 21968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:38.993269 21973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:38.993351 21974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:38.993351 21976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:38.994261 21968 server_base.cc:1034] running on GCE node
I20260621 02:43:38.994995 21968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:38.996315 21968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:38.997527 21968 hybrid_clock.cc:648] HybridClock initialized: now 1782009818997496 us; error 47 us; skew 500 ppm
I20260621 02:43:38.997870 21968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:38 dist-test-slave-jkn7 krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:39.004251 21968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:43:39.005826 21968 webserver.cc:460] Webserver started at http://127.0.0.1:40147/ using document root <none> and password file <none>
I20260621 02:43:39.006517 21968 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:39.006657 21968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:39.006953 21968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:39.009900 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "d7164ba8876c48bea53cc7334f812288"
format_stamp: "Formatted at 2026-06-21 02:43:39 on dist-test-slave-jkn7"
server_key: "819bf626b8f532d5aa0d89eeb4b68d74"
server_key_iv: "a2efd3d685569de3237b1771e4036352"
server_key_version: "encryptionkey@0"
I20260621 02:43:39.010650 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "d7164ba8876c48bea53cc7334f812288"
format_stamp: "Formatted at 2026-06-21 02:43:39 on dist-test-slave-jkn7"
server_key: "819bf626b8f532d5aa0d89eeb4b68d74"
server_key_iv: "a2efd3d685569de3237b1771e4036352"
server_key_version: "encryptionkey@0"
I20260621 02:43:39.015745 21968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:43:39.019124 21983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:39.020382 21968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:43:39.020560 21968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "d7164ba8876c48bea53cc7334f812288"
format_stamp: "Formatted at 2026-06-21 02:43:39 on dist-test-slave-jkn7"
server_key: "819bf626b8f532d5aa0d89eeb4b68d74"
server_key_iv: "a2efd3d685569de3237b1771e4036352"
server_key_version: "encryptionkey@0"
I20260621 02:43:39.020758 21968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:39.026453 21968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:39.030707 21968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:39.030983 21968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:39.049391 21968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34037
I20260621 02:43:39.049469 22034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34037 every 8 connection(s)
I20260621 02:43:39.050763 21968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:39.054288 22035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:39.054744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 21968
I20260621 02:43:39.054926 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:39.055346 388 external_mini_cluster.cc:1428] Setting key abb1dc0c92df18ff8027a3c49e9ca75e
I20260621 02:43:39.064256 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:39.057070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:54895 (local address 127.0.0.1:34037)
0621 02:43:39.057894 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:39.057911 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:39.057962 (+ 51us) server_negotiation.cc:407] Connection header received
0621 02:43:39.060711 (+ 2749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:39.060737 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:39.061159 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:39.061600 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:39.062492 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:39.062500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:39.062516 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:39.062531 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:39.062590 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:39.062612 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:39.062817 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:39.062937 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 02:43:39.063110 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":313,"threads_started":1}
I20260621 02:43:39.067828 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:39.068446 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288: Bootstrap starting.
I20260621 02:43:39.071542 22035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:39.072860 22035 log.cc:824] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:39.074402 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:39.072037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:43627 (local address 127.0.0.1:34037)
0621 02:43:39.072195 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:39.072201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:39.072491 (+ 290us) server_negotiation.cc:407] Connection header received
0621 02:43:39.072612 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:39.072619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:39.072695 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:39.072795 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:39.073724 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:39.073729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:39.073734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:39.073756 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:39.073790 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:39.073805 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:39.073926 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:39.074044 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 02:43:39.074103 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:43:39.076200 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288: No bootstrap required, opened a new log
I20260621 02:43:39.083527 22035 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7164ba8876c48bea53cc7334f812288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34037 } }
I20260621 02:43:39.083853 22035 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:39.083925 22035 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7164ba8876c48bea53cc7334f812288, State: Initialized, Role: FOLLOWER
I20260621 02:43:39.084470 22035 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7164ba8876c48bea53cc7334f812288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34037 } }
I20260621 02:43:39.084669 22035 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:39.084777 22035 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:39.084897 22035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:39.087033 22035 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7164ba8876c48bea53cc7334f812288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34037 } }
I20260621 02:43:39.087585 22035 leader_election.cc:302] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7164ba8876c48bea53cc7334f812288; no voters:
I20260621 02:43:39.088285 22035 leader_election.cc:288] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:39.088366 22045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:39.089063 22045 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [term 1 LEADER]: Becoming Leader. State: Replica: d7164ba8876c48bea53cc7334f812288, State: Running, Role: LEADER
I20260621 02:43:39.089567 22045 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7164ba8876c48bea53cc7334f812288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34037 } }
I20260621 02:43:39.090363 22035 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:39.092240 22046 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7164ba8876c48bea53cc7334f812288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7164ba8876c48bea53cc7334f812288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34037 } } }
I20260621 02:43:39.092698 22046 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:39.093128 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7164ba8876c48bea53cc7334f812288. Latest consensus state: current_term: 1 leader_uuid: "d7164ba8876c48bea53cc7334f812288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7164ba8876c48bea53cc7334f812288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34037 } } }
I20260621 02:43:39.093447 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:39.094933 22051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:39.100082 22051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:39.109018 22051 catalog_manager.cc:1347] Generated new cluster ID: c720fc694d2641e3b4a2497899b7d174
I20260621 02:43:39.109141 22051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:39.146986 22051 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:39.147497 22051 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:39.154888 22051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7164ba8876c48bea53cc7334f812288: Generated new TSK 0
I20260621 02:43:39.155848 22051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:39.169163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21968
W20260621 02:43:39.170197 22030 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.12:54895
W20260621 02:43:39.170871 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 21968: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:39.171008 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 21968
2026-06-21T02:43:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:39 dist-test-slave-jkn7 krb5kdc[22066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:39 dist-test-slave-jkn7 krb5kdc[22066](info): set up 2 sockets
Jun 21 02:43:39 dist-test-slave-jkn7 krb5kdc[22066](info): commencing operation
krb5kdc: starting...
W20260621 02:43:40.269441 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.009s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:40 dist-test-slave-jkn7 krb5kdc[22066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:40.508690 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40801
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:40.828012 22079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:40.828320 22079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:40.828498 22079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:40.836207 22079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:40.836334 22079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:40.836421 22079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:40.836531 22079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:40.836611 22079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:40.846884 22079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:40.848282 22079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:40.849663 22079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:40.863234 22084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:40.864135 22079 server_base.cc:1034] running on GCE node
W20260621 02:43:40.864225 22087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:40.864261 22085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:40.865185 22079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:40.866766 22079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:40.868006 22079 hybrid_clock.cc:648] HybridClock initialized: now 1782009820867955 us; error 43 us; skew 500 ppm
I20260621 02:43:40.868379 22079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:40 dist-test-slave-jkn7 krb5kdc[22066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:40.871618 22079 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:40.873358 22079 webserver.cc:460] Webserver started at http://127.0.97.62:44445/ using document root <none> and password file <none>
I20260621 02:43:40.874082 22079 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:40.874228 22079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:40.874542 22079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:40.877554 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "afe72a35672e44a2936381bc405a9958"
format_stamp: "Formatted at 2026-06-21 02:43:40 on dist-test-slave-jkn7"
server_key: "3f3213690211b40895f77792a83affdf"
server_key_iv: "969f79ef4132ed7c45b6a28db9a5d7e1"
server_key_version: "encryptionkey@0"
I20260621 02:43:40.878330 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "afe72a35672e44a2936381bc405a9958"
format_stamp: "Formatted at 2026-06-21 02:43:40 on dist-test-slave-jkn7"
server_key: "3f3213690211b40895f77792a83affdf"
server_key_iv: "969f79ef4132ed7c45b6a28db9a5d7e1"
server_key_version: "encryptionkey@0"
I20260621 02:43:40.883370 22079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260621 02:43:40.886795 22094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:40.888011 22079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:43:40.888190 22079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "afe72a35672e44a2936381bc405a9958"
format_stamp: "Formatted at 2026-06-21 02:43:40 on dist-test-slave-jkn7"
server_key: "3f3213690211b40895f77792a83affdf"
server_key_iv: "969f79ef4132ed7c45b6a28db9a5d7e1"
server_key_version: "encryptionkey@0"
I20260621 02:43:40.888376 22079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:40.904968 22079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:40.908901 22079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:40.909198 22079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:40.927574 22079 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40801
I20260621 02:43:40.927608 22145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40801 every 8 connection(s)
I20260621 02:43:40.928911 22079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:40.930480 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 22079
I20260621 02:43:40.930662 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:40.931188 388 external_mini_cluster.cc:1428] Setting key 15183943283b9e22bfdd5db88210d5f5
I20260621 02:43:40.933470 22146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:43:40 dist-test-slave-jkn7 krb5kdc[22066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:40.946856 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958: Bootstrap starting.
I20260621 02:43:40.950127 22146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:40.951225 22146 log.cc:824] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:40.954315 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958: No bootstrap required, opened a new log
I20260621 02:43:40.954347 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:40.933000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36411 (local address 127.0.97.62:40801)
0621 02:43:40.933794 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:40.933806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:40.933845 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:43:40.935593 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:40.935616 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:40.935994 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:40.936329 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:40.938081 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:40.939437 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:40.941240 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:40.942008 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:40.942139 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:40.946924 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:40.946932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:40.946946 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:40.947005 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:40.950141 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:40.950940 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:40.950947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:40.950953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:40.951119 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:40.951637 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:40.951647 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:40.951649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:40.952177 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:40.952519 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:40.953027 (+ 508us) server_negotiation.cc:299] Negotiation successful
0621 02:43:40.953403 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":373,"threads_started":1}
I20260621 02:43:40.958130 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:40.962412 22146 raft_consensus.cc:348] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe72a35672e44a2936381bc405a9958" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40801 } }
I20260621 02:43:40.962800 22146 raft_consensus.cc:374] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:40.962920 22146 raft_consensus.cc:729] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe72a35672e44a2936381bc405a9958, State: Initialized, Role: FOLLOWER
I20260621 02:43:40.963537 22146 consensus_queue.cc:260] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe72a35672e44a2936381bc405a9958" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40801 } }
I20260621 02:43:40.963773 22146 raft_consensus.cc:388] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:40.963927 22146 raft_consensus.cc:482] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:40.964089 22146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:40.966885 22146 raft_consensus.cc:504] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe72a35672e44a2936381bc405a9958" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40801 } }
I20260621 02:43:40.967408 22146 leader_election.cc:302] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afe72a35672e44a2936381bc405a9958; no voters:
I20260621 02:43:40.968031 22146 leader_election.cc:288] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:40.968129 22156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:40.968793 22156 raft_consensus.cc:686] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [term 1 LEADER]: Becoming Leader. State: Replica: afe72a35672e44a2936381bc405a9958, State: Running, Role: LEADER
I20260621 02:43:40.969718 22156 consensus_queue.cc:237] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe72a35672e44a2936381bc405a9958" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40801 } }
I20260621 02:43:40.970433 22146 sys_catalog.cc:564] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:40.972106 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afe72a35672e44a2936381bc405a9958. Latest consensus state: current_term: 1 leader_uuid: "afe72a35672e44a2936381bc405a9958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe72a35672e44a2936381bc405a9958" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40801 } } }
I20260621 02:43:40.972745 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:40.975093 22158 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afe72a35672e44a2936381bc405a9958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe72a35672e44a2936381bc405a9958" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40801 } } }
I20260621 02:43:40.975469 22158 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:40.978828 22161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:40.980135 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:40.962253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38377 (local address 127.0.97.62:40801)
0621 02:43:40.962445 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:40.962451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:40.962843 (+ 392us) server_negotiation.cc:407] Connection header received
0621 02:43:40.963021 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:40.963027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:40.963102 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:40.963211 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:40.964805 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:40.966050 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:40.967305 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:40.968081 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:40.968219 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:40.971188 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:40.971203 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:40.971209 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:40.971242 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:40.975946 (+ 4704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:40.977195 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:40.977204 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:40.977210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:40.977529 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:40.978173 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:40.978179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:40.978188 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:40.978578 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:40.978883 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:40.979274 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 02:43:40.979439 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:43:40.985682 22161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:40.994637 22161 catalog_manager.cc:1347] Generated new cluster ID: a98b60ec5f1a4830bc88abf95d1c7294
I20260621 02:43:40.994776 22161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:41.039796 22161 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:41.041128 22161 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:41.051712 22161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afe72a35672e44a2936381bc405a9958: Generated new TSK 0
I20260621 02:43:41.052480 22161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:41.060534 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22079
I20260621 02:43:41.061590 22141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:41.153927 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 22079
W20260621 02:43:41.170979 22064 connection.cc:535] client connection to 127.0.97.62:40801 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40801): Connection reset by peer (error 104)
2026-06-21T02:43:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:41 dist-test-slave-jkn7 krb5kdc[22178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:41 dist-test-slave-jkn7 krb5kdc[22178](info): set up 2 sockets
Jun 21 02:43:41 dist-test-slave-jkn7 krb5kdc[22178](info): commencing operation
krb5kdc: starting...
W20260621 02:43:42.253765 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:42 dist-test-slave-jkn7 krb5kdc[22178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:42.516336 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46781
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:42.843026 22191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:42.843307 22191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:42.843487 22191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:42.850792 22191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:42.850893 22191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:42.851001 22191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:42.851087 22191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:42.851153 22191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:42.860859 22191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:42.862099 22191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:42.863384 22191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:42.875976 22196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:42.875928 22197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:42.875998 22199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:42.876956 22191 server_base.cc:1034] running on GCE node
I20260621 02:43:42.877840 22191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:42.879313 22191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:42.880550 22191 hybrid_clock.cc:648] HybridClock initialized: now 1782009822880489 us; error 31 us; skew 500 ppm
I20260621 02:43:42.880952 22191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:42 dist-test-slave-jkn7 krb5kdc[22178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:42.884279 22191 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:42.885938 22191 webserver.cc:460] Webserver started at http://127.0.97.62:38977/ using document root <none> and password file <none>
I20260621 02:43:42.886664 22191 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:42.886816 22191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:42.887224 22191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:42.890450 22191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "76485ebe552d40239771cb7752cf5e00"
format_stamp: "Formatted at 2026-06-21 02:43:42 on dist-test-slave-jkn7"
server_key: "59b649857e70b1437ba1f5326d3d02a4"
server_key_iv: "fb354c44014a8ec8d243c5b0b82abe1b"
server_key_version: "encryptionkey@0"
I20260621 02:43:42.891268 22191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "76485ebe552d40239771cb7752cf5e00"
format_stamp: "Formatted at 2026-06-21 02:43:42 on dist-test-slave-jkn7"
server_key: "59b649857e70b1437ba1f5326d3d02a4"
server_key_iv: "fb354c44014a8ec8d243c5b0b82abe1b"
server_key_version: "encryptionkey@0"
I20260621 02:43:42.896944 22191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:43:42.900651 22206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:42.902102 22191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:43:42.902302 22191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "76485ebe552d40239771cb7752cf5e00"
format_stamp: "Formatted at 2026-06-21 02:43:42 on dist-test-slave-jkn7"
server_key: "59b649857e70b1437ba1f5326d3d02a4"
server_key_iv: "fb354c44014a8ec8d243c5b0b82abe1b"
server_key_version: "encryptionkey@0"
I20260621 02:43:42.902527 22191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:42.925945 22191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:42.930529 22191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:42.930883 22191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:42.949612 22191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46781
I20260621 02:43:42.949642 22257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46781 every 8 connection(s)
I20260621 02:43:42.950975 22191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:42.954540 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:42.959785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 22191
I20260621 02:43:42.959960 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:42.960464 388 external_mini_cluster.cc:1428] Setting key 739c63af545a9b69518bdf184717288e
I20260621 02:43:42.965433 22258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00: Bootstrap starting.
I20260621 02:43:42.968372 22258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:42.969427 22258 log.cc:824] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:42.972141 22258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00: No bootstrap required, opened a new log
I20260621 02:43:42.973387 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:42.962290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55691 (local address 127.0.97.62:46781)
0621 02:43:42.963158 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:42.963169 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:42.963210 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:43:42.964852 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:42.964873 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:42.965241 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:42.965701 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:42.966217 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:42.967517 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:42.968895 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:42.969678 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:42.969863 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:42.971120 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:42.971128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:42.971143 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:42.971653 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:42.971724 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:42.971753 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:42.971917 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:42.972201 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 02:43:42.972375 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":430,"threads_started":1}
I20260621 02:43:42.977249 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:42.980445 22258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76485ebe552d40239771cb7752cf5e00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } }
I20260621 02:43:42.980881 22258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:42.980989 22258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76485ebe552d40239771cb7752cf5e00, State: Initialized, Role: FOLLOWER
I20260621 02:43:42.981691 22258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76485ebe552d40239771cb7752cf5e00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } }
I20260621 02:43:42.982020 22258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:42.982187 22258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:42.982362 22258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:42.984715 22258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76485ebe552d40239771cb7752cf5e00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } }
I20260621 02:43:42.985239 22258 leader_election.cc:302] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76485ebe552d40239771cb7752cf5e00; no voters:
I20260621 02:43:42.985949 22258 leader_election.cc:288] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:42.986094 22268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:42.986783 22268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [term 1 LEADER]: Becoming Leader. State: Replica: 76485ebe552d40239771cb7752cf5e00, State: Running, Role: LEADER
I20260621 02:43:42.987409 22268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76485ebe552d40239771cb7752cf5e00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } }
I20260621 02:43:42.989763 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:42.981749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34233 (local address 127.0.97.62:46781)
0621 02:43:42.981922 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:42.981927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:42.982552 (+ 625us) server_negotiation.cc:407] Connection header received
0621 02:43:42.982769 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:42.982774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:42.982851 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:42.982972 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:42.983471 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:42.984252 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:42.985860 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:42.986777 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:42.986961 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:42.988712 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:42.988718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:42.988723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:42.988749 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:42.988826 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:42.988855 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:42.988997 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:42.989178 (+ 181us) server_negotiation.cc:299] Negotiation successful
0621 02:43:42.989252 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:43:42.990239 22258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:42.991322 22269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76485ebe552d40239771cb7752cf5e00. Latest consensus state: current_term: 1 leader_uuid: "76485ebe552d40239771cb7752cf5e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76485ebe552d40239771cb7752cf5e00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } } }
I20260621 02:43:42.991755 22269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:42.993259 22270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76485ebe552d40239771cb7752cf5e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76485ebe552d40239771cb7752cf5e00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46781 } } }
I20260621 02:43:42.993662 22270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:42.994696 22273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:43.000849 22273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:43.009850 22273 catalog_manager.cc:1347] Generated new cluster ID: 3eda765754b0431c8dcf9581704bb062
I20260621 02:43:43.009994 22273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:43.031824 22273 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:43.033008 22273 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:43.040160 22273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76485ebe552d40239771cb7752cf5e00: Generated new TSK 0
I20260621 02:43:43.040891 22273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:43.056033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22191
W20260621 02:43:43.057036 22253 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55691
W20260621 02:43:43.057771 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22191: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:43.057915 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 22191
W20260621 02:43:43.074669 22176 connection.cc:535] client connection to 127.0.97.62:46781 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:46781): Connection reset by peer (error 104)
2026-06-21T02:43:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:43 dist-test-slave-jkn7 krb5kdc[22289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:43 dist-test-slave-jkn7 krb5kdc[22289](info): set up 2 sockets
Jun 21 02:43:43 dist-test-slave-jkn7 krb5kdc[22289](info): commencing operation
krb5kdc: starting...
W20260621 02:43:44.157721 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:44 dist-test-slave-jkn7 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:44.413789 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38277
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:44.732303 22302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:44.732553 22302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:44.732865 22302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:44.739856 22302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:44.739953 22302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:44.740077 22302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:44.740180 22302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:44.740262 22302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:44.749593 22302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:44.750735 22302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:44.751943 22302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:44.763020 22307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:44.762976 22310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:44.763058 22308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:44.763979 22302 server_base.cc:1034] running on GCE node
I20260621 02:43:44.764715 22302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:44.766013 22302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:44.767251 22302 hybrid_clock.cc:648] HybridClock initialized: now 1782009824767209 us; error 54 us; skew 500 ppm
I20260621 02:43:44.767599 22302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:44 dist-test-slave-jkn7 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:44.775277 22302 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:44.776690 22302 webserver.cc:460] Webserver started at http://127.0.97.62:46141/ using document root <none> and password file <none>
I20260621 02:43:44.777405 22302 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:44.777561 22302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:44.777853 22302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:44.780629 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "452a251dd347438eb67d165da99bca62"
format_stamp: "Formatted at 2026-06-21 02:43:44 on dist-test-slave-jkn7"
server_key: "f093f09219f25e553883db66be1435bf"
server_key_iv: "c6eba0f753deeb29c338be470c7fc6a0"
server_key_version: "encryptionkey@0"
I20260621 02:43:44.781437 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "452a251dd347438eb67d165da99bca62"
format_stamp: "Formatted at 2026-06-21 02:43:44 on dist-test-slave-jkn7"
server_key: "f093f09219f25e553883db66be1435bf"
server_key_iv: "c6eba0f753deeb29c338be470c7fc6a0"
server_key_version: "encryptionkey@0"
I20260621 02:43:44.786190 22302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 02:43:44.789333 22317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:44.790549 22302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 02:43:44.790715 22302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "452a251dd347438eb67d165da99bca62"
format_stamp: "Formatted at 2026-06-21 02:43:44 on dist-test-slave-jkn7"
server_key: "f093f09219f25e553883db66be1435bf"
server_key_iv: "c6eba0f753deeb29c338be470c7fc6a0"
server_key_version: "encryptionkey@0"
I20260621 02:43:44.790906 22302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:44.796962 22302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:44.800678 22302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:44.800957 22302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:44.818782 22302 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38277
I20260621 02:43:44.818859 22368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38277 every 8 connection(s)
I20260621 02:43:44.820065 22302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:44.823506 22369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:44.826188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 22302
I20260621 02:43:44.826344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:44.826794 388 external_mini_cluster.cc:1428] Setting key dab9dab833d8747f12a9f14c943e1f95
I20260621 02:43:44.834102 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:44.828438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53537 (local address 127.0.97.62:38277)
0621 02:43:44.829188 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:44.829199 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:44.829238 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:43:44.830842 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:44.830863 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:44.831196 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:44.831570 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:44.832575 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:44.832581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:44.832592 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:44.832604 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:44.832662 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:44.832682 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:44.832931 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:44.833075 (+ 144us) server_negotiation.cc:299] Negotiation successful
0621 02:43:44.833235 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":335,"threads_started":1}
I20260621 02:43:44.834976 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62: Bootstrap starting.
I20260621 02:43:44.837512 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:44.838241 22369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:44.839192 22369 log.cc:824] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:44.842353 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62: No bootstrap required, opened a new log
I20260621 02:43:44.843981 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:44.841504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52995 (local address 127.0.97.62:38277)
0621 02:43:44.841644 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:44.841647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:44.842103 (+ 456us) server_negotiation.cc:407] Connection header received
0621 02:43:44.842280 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:44.842285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:44.842336 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:44.842436 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:44.843359 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:44.843365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:44.843370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:44.843386 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:44.843429 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:44.843443 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:44.843566 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:44.843679 (+ 113us) server_negotiation.cc:299] Negotiation successful
0621 02:43:44.843731 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:43:44.850106 22369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a251dd347438eb67d165da99bca62" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38277 } }
I20260621 02:43:44.850569 22369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:44.850677 22369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452a251dd347438eb67d165da99bca62, State: Initialized, Role: FOLLOWER
I20260621 02:43:44.851217 22369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a251dd347438eb67d165da99bca62" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38277 } }
I20260621 02:43:44.851459 22369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:44.851600 22369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:44.851770 22369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:44.854167 22369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a251dd347438eb67d165da99bca62" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38277 } }
I20260621 02:43:44.854688 22369 leader_election.cc:302] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 452a251dd347438eb67d165da99bca62; no voters:
I20260621 02:43:44.855160 22369 leader_election.cc:288] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:44.855305 22379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:44.855938 22379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [term 1 LEADER]: Becoming Leader. State: Replica: 452a251dd347438eb67d165da99bca62, State: Running, Role: LEADER
I20260621 02:43:44.856464 22379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a251dd347438eb67d165da99bca62" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38277 } }
I20260621 02:43:44.857092 22369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:44.859253 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 452a251dd347438eb67d165da99bca62. Latest consensus state: current_term: 1 leader_uuid: "452a251dd347438eb67d165da99bca62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a251dd347438eb67d165da99bca62" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38277 } } }
I20260621 02:43:44.859751 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:44.860313 22380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "452a251dd347438eb67d165da99bca62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a251dd347438eb67d165da99bca62" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38277 } } }
I20260621 02:43:44.860643 22380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:44.866379 22386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:44.871421 22386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:44.880637 22386 catalog_manager.cc:1347] Generated new cluster ID: 5a2d86f2fbe54806b4d8228f621714c3
I20260621 02:43:44.880757 22386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:44.905779 22386 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:44.906208 22386 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:44.919625 22386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 452a251dd347438eb67d165da99bca62: Generated new TSK 0
I20260621 02:43:44.920360 22386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:44.937477 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22302
W20260621 02:43:44.938632 22364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53537
W20260621 02:43:44.939283 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22302: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:44.939430 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 22302
2026-06-21T02:43:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:45 dist-test-slave-jkn7 krb5kdc[22400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:45 dist-test-slave-jkn7 krb5kdc[22400](info): set up 2 sockets
Jun 21 02:43:45 dist-test-slave-jkn7 krb5kdc[22400](info): commencing operation
krb5kdc: starting...
W20260621 02:43:46.053690 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:46 dist-test-slave-jkn7 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:46.300395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36505
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:46.614641 22413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:46.614975 22413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:46.615135 22413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:46.622354 22413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:46.622505 22413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:46.622604 22413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:46.622725 22413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:46.622802 22413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:46.632345 22413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:46.633596 22413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:46.634857 22413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:46.646433 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:46.646394 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:46.646358 22421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:46.647264 22413 server_base.cc:1034] running on GCE node
I20260621 02:43:46.648034 22413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:46.649457 22413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:46.650656 22413 hybrid_clock.cc:648] HybridClock initialized: now 1782009826650601 us; error 47 us; skew 500 ppm
I20260621 02:43:46.650983 22413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:46 dist-test-slave-jkn7 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:46.654130 22413 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:46.655616 22413 webserver.cc:460] Webserver started at http://127.0.97.62:44981/ using document root <none> and password file <none>
I20260621 02:43:46.656222 22413 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:46.656319 22413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:46.656556 22413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:46.659380 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "7f99c78411cc4705840abb9466c409a2"
format_stamp: "Formatted at 2026-06-21 02:43:46 on dist-test-slave-jkn7"
server_key: "68c697453979ed21827b6675fd4ae789"
server_key_iv: "17ada187c5ba87a021d239801cca939f"
server_key_version: "encryptionkey@0"
I20260621 02:43:46.660071 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "7f99c78411cc4705840abb9466c409a2"
format_stamp: "Formatted at 2026-06-21 02:43:46 on dist-test-slave-jkn7"
server_key: "68c697453979ed21827b6675fd4ae789"
server_key_iv: "17ada187c5ba87a021d239801cca939f"
server_key_version: "encryptionkey@0"
I20260621 02:43:46.664983 22413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:43:46.668291 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:46.669621 22413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:43:46.669811 22413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "7f99c78411cc4705840abb9466c409a2"
format_stamp: "Formatted at 2026-06-21 02:43:46 on dist-test-slave-jkn7"
server_key: "68c697453979ed21827b6675fd4ae789"
server_key_iv: "17ada187c5ba87a021d239801cca939f"
server_key_version: "encryptionkey@0"
I20260621 02:43:46.670002 22413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:46.685025 22413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:46.689261 22413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:46.689601 22413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:46.708084 22413 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36505
I20260621 02:43:46.708117 22479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36505 every 8 connection(s)
I20260621 02:43:46.709416 22413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:46.712880 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:46.713447 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 22413
I20260621 02:43:46.713632 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:46.714049 388 external_mini_cluster.cc:1428] Setting key 42ecbd6f1353c70ba8514c5fd760cda3
I20260621 02:43:46.726595 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2: Bootstrap starting.
Jun 21 02:43:46 dist-test-slave-jkn7 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:46.730276 22480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:46.731565 22480 log.cc:824] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:46.734251 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2: No bootstrap required, opened a new log
I20260621 02:43:46.738039 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:46.715796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39411 (local address 127.0.97.62:36505)
0621 02:43:46.716606 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:46.716628 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:46.716680 (+ 52us) server_negotiation.cc:407] Connection header received
0621 02:43:46.719339 (+ 2659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:46.719367 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:46.719863 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:46.720282 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:46.721637 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:46.723569 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:46.724838 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:46.725830 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:46.725999 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:46.730765 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:46.730777 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:46.730793 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:46.730834 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:46.734069 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:46.734786 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:46.734793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:46.734798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:46.734957 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:46.735335 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:46.735341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:46.735344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:46.735853 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:46.736165 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:46.736632 (+ 467us) server_negotiation.cc:299] Negotiation successful
0621 02:43:46.737030 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":346,"threads_started":1}
I20260621 02:43:46.741477 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:46.742048 22480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99c78411cc4705840abb9466c409a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36505 } }
I20260621 02:43:46.742384 22480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:46.742483 22480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f99c78411cc4705840abb9466c409a2, State: Initialized, Role: FOLLOWER
I20260621 02:43:46.743041 22480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99c78411cc4705840abb9466c409a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36505 } }
I20260621 02:43:46.743281 22480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:46.743412 22480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:46.743580 22480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:46.746312 22480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99c78411cc4705840abb9466c409a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36505 } }
I20260621 02:43:46.746845 22480 leader_election.cc:302] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f99c78411cc4705840abb9466c409a2; no voters:
I20260621 02:43:46.747623 22480 leader_election.cc:288] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:46.747727 22490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:46.750042 22480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:46.753576 22490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [term 1 LEADER]: Becoming Leader. State: Replica: 7f99c78411cc4705840abb9466c409a2, State: Running, Role: LEADER
I20260621 02:43:46.754366 22490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99c78411cc4705840abb9466c409a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36505 } }
I20260621 02:43:46.757429 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f99c78411cc4705840abb9466c409a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99c78411cc4705840abb9466c409a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36505 } } }
I20260621 02:43:46.757772 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:46.758737 22498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:46.761049 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:46.745802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33131 (local address 127.0.97.62:36505)
0621 02:43:46.745975 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:46.745982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:46.746406 (+ 424us) server_negotiation.cc:407] Connection header received
0621 02:43:46.746598 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:46.746606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:46.746687 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:46.746785 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:46.748439 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:46.749239 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:46.750665 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:46.751108 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:46.751257 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:46.753732 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:46.753737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:46.753745 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:46.753781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:46.756947 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:46.757905 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:46.757918 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:46.757922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:46.758068 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:46.758542 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:46.758546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:46.758551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:46.759031 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:46.759283 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:46.760293 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0621 02:43:46.760513 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:43:46.762571 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f99c78411cc4705840abb9466c409a2. Latest consensus state: current_term: 1 leader_uuid: "7f99c78411cc4705840abb9466c409a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99c78411cc4705840abb9466c409a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36505 } } }
I20260621 02:43:46.762820 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:46.768108 22498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:46.777582 22498 catalog_manager.cc:1347] Generated new cluster ID: d528931545604574bbe4a7bfd0813046
I20260621 02:43:46.777689 22498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:46.795915 22498 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:46.797101 22498 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:46.811794 22498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f99c78411cc4705840abb9466c409a2: Generated new TSK 0
I20260621 02:43:46.812879 22498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:46.828616 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22413
I20260621 02:43:46.829778 22475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:46.923437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 22413
2026-06-21T02:43:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:47 dist-test-slave-jkn7 krb5kdc[22512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:47 dist-test-slave-jkn7 krb5kdc[22512](info): set up 2 sockets
Jun 21 02:43:47 dist-test-slave-jkn7 krb5kdc[22512](info): commencing operation
krb5kdc: starting...
W20260621 02:43:48.013556 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:48 dist-test-slave-jkn7 krb5kdc[22512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:48.254076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34491
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:48.574054 22525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:48.574393 22525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:48.574572 22525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:48.582297 22525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:48.582482 22525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:48.582573 22525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:48.582667 22525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:48.582749 22525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:48.592422 22525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:48.593834 22525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:48.595197 22525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:48.607467 22530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:48.607430 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:48.607401 22531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:48.608229 22525 server_base.cc:1034] running on GCE node
I20260621 02:43:48.609045 22525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:48.610584 22525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:48.611840 22525 hybrid_clock.cc:648] HybridClock initialized: now 1782009828611803 us; error 58 us; skew 500 ppm
I20260621 02:43:48.612232 22525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:48 dist-test-slave-jkn7 krb5kdc[22512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:48.615332 22525 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:48.616845 22525 webserver.cc:460] Webserver started at http://127.0.97.62:35595/ using document root <none> and password file <none>
I20260621 02:43:48.617645 22525 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:48.617789 22525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:48.618095 22525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:48.621054 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "ff33949ef64349b4be9fdd36c94fbd89"
format_stamp: "Formatted at 2026-06-21 02:43:48 on dist-test-slave-jkn7"
server_key: "8703b1e91c230738c279f62ad6d890c7"
server_key_iv: "b66bc13341289fa431d0e67d02ba06e8"
server_key_version: "encryptionkey@0"
I20260621 02:43:48.621872 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff33949ef64349b4be9fdd36c94fbd89"
format_stamp: "Formatted at 2026-06-21 02:43:48 on dist-test-slave-jkn7"
server_key: "8703b1e91c230738c279f62ad6d890c7"
server_key_iv: "b66bc13341289fa431d0e67d02ba06e8"
server_key_version: "encryptionkey@0"
I20260621 02:43:48.627048 22525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:43:48.630420 22540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:48.631762 22525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:43:48.631942 22525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "ff33949ef64349b4be9fdd36c94fbd89"
format_stamp: "Formatted at 2026-06-21 02:43:48 on dist-test-slave-jkn7"
server_key: "8703b1e91c230738c279f62ad6d890c7"
server_key_iv: "b66bc13341289fa431d0e67d02ba06e8"
server_key_version: "encryptionkey@0"
I20260621 02:43:48.632146 22525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:48.658807 22525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:48.662870 22525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:48.663157 22525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:48.681562 22525 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34491
I20260621 02:43:48.681595 22591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34491 every 8 connection(s)
I20260621 02:43:48.682986 22525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:48.686782 22592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:48.687471 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 22525
I20260621 02:43:48.687667 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:48.688208 388 external_mini_cluster.cc:1428] Setting key ad299bc336092d12e853dc00fcf2baed
I20260621 02:43:48.699653 22592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89: Bootstrap starting.
I20260621 02:43:48.701959 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:48.690159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53191 (local address 127.0.97.62:34491)
0621 02:43:48.691135 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:48.691147 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:48.691192 (+ 45us) server_negotiation.cc:407] Connection header received
0621 02:43:48.693605 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:48.693627 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:48.694013 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:48.694407 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:48.694893 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:48.696119 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:48.697429 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:48.698123 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:48.698312 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:48.699528 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:48.699535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:48.699548 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:48.700266 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:48.700346 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:48.700371 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:48.700514 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:48.700698 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 02:43:48.700967 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":395,"threads_started":1}
I20260621 02:43:48.702797 22592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:48.703810 22592 log.cc:824] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:48.705516 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:48.706637 22592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89: No bootstrap required, opened a new log
I20260621 02:43:48.714393 22592 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff33949ef64349b4be9fdd36c94fbd89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } }
I20260621 02:43:48.714818 22592 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:48.714946 22592 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff33949ef64349b4be9fdd36c94fbd89, State: Initialized, Role: FOLLOWER
I20260621 02:43:48.715595 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:48.709687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52673 (local address 127.0.97.62:34491)
0621 02:43:48.709886 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:48.709894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:48.710252 (+ 358us) server_negotiation.cc:407] Connection header received
0621 02:43:48.710480 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:48.710483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:48.710538 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:48.710637 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:48.711160 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:48.711877 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:48.713148 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:48.713685 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:48.713822 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:48.714851 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:48.714856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:48.714860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:48.714882 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:48.714953 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:48.714975 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:48.715057 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:48.715215 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 02:43:48.715305 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:43:48.715620 22592 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff33949ef64349b4be9fdd36c94fbd89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } }
I20260621 02:43:48.716013 22592 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:48.716161 22592 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:48.716343 22592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:48.718619 22592 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff33949ef64349b4be9fdd36c94fbd89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } }
I20260621 02:43:48.719092 22592 leader_election.cc:302] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff33949ef64349b4be9fdd36c94fbd89; no voters:
I20260621 02:43:48.719653 22592 leader_election.cc:288] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:48.719780 22602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:48.721313 22602 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [term 1 LEADER]: Becoming Leader. State: Replica: ff33949ef64349b4be9fdd36c94fbd89, State: Running, Role: LEADER
I20260621 02:43:48.721827 22592 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:48.721848 22602 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff33949ef64349b4be9fdd36c94fbd89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } }
I20260621 02:43:48.725180 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff33949ef64349b4be9fdd36c94fbd89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff33949ef64349b4be9fdd36c94fbd89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } }
I20260621 02:43:48.725644 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff33949ef64349b4be9fdd36c94fbd89. Latest consensus state: current_term: 1 leader_uuid: "ff33949ef64349b4be9fdd36c94fbd89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff33949ef64349b4be9fdd36c94fbd89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } }
I20260621 02:43:48.726189 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:48.726186 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:48.734207 22613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:48.738391 22613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:48.746088 22613 catalog_manager.cc:1347] Generated new cluster ID: e730295140f04b1e9d77a69d2527c949
I20260621 02:43:48.746198 22613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:48.755884 22613 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:48.757494 22613 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:48.774165 22613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff33949ef64349b4be9fdd36c94fbd89: Generated new TSK 0
I20260621 02:43:48.774950 22613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:48.781371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22525
W20260621 02:43:48.782341 22587 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53191
W20260621 02:43:48.782974 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22525: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:48.783115 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 22525
2026-06-21T02:43:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:48 dist-test-slave-jkn7 krb5kdc[22623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:48 dist-test-slave-jkn7 krb5kdc[22623](info): set up 2 sockets
Jun 21 02:43:48 dist-test-slave-jkn7 krb5kdc[22623](info): commencing operation
krb5kdc: starting...
W20260621 02:43:49.885671 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:50 dist-test-slave-jkn7 krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:50.138178 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46337
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:50.453058 22636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:50.453373 22636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:50.453482 22636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:50.460507 22636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:50.460601 22636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:50.460646 22636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:50.460706 22636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:50.460750 22636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:50.470149 22636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:50.471231 22636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:50.472378 22636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:50.483871 22642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:50.483817 22644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:50.484275 22641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:50.486569 22636 server_base.cc:1034] running on GCE node
I20260621 02:43:50.487339 22636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:50.488690 22636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:50.489912 22636 hybrid_clock.cc:648] HybridClock initialized: now 1782009830489881 us; error 44 us; skew 500 ppm
I20260621 02:43:50.490245 22636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:50 dist-test-slave-jkn7 krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:50.497709 22636 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:50.499258 22636 webserver.cc:460] Webserver started at http://127.0.97.62:42471/ using document root <none> and password file <none>
I20260621 02:43:50.499950 22636 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:50.500097 22636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:50.500414 22636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:50.503348 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "7ecbd9e59d804ac597be24c8a5c94532"
format_stamp: "Formatted at 2026-06-21 02:43:50 on dist-test-slave-jkn7"
server_key: "f36be5aa1236a026c45d48ef82e68dd8"
server_key_iv: "7a5daf9b5bd8fe5bdaf843517371c87d"
server_key_version: "encryptionkey@0"
I20260621 02:43:50.504104 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "7ecbd9e59d804ac597be24c8a5c94532"
format_stamp: "Formatted at 2026-06-21 02:43:50 on dist-test-slave-jkn7"
server_key: "f36be5aa1236a026c45d48ef82e68dd8"
server_key_iv: "7a5daf9b5bd8fe5bdaf843517371c87d"
server_key_version: "encryptionkey@0"
I20260621 02:43:50.509076 22636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 02:43:50.512351 22651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:50.513689 22636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:43:50.513871 22636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "7ecbd9e59d804ac597be24c8a5c94532"
format_stamp: "Formatted at 2026-06-21 02:43:50 on dist-test-slave-jkn7"
server_key: "f36be5aa1236a026c45d48ef82e68dd8"
server_key_iv: "7a5daf9b5bd8fe5bdaf843517371c87d"
server_key_version: "encryptionkey@0"
I20260621 02:43:50.514071 22636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:50.519753 22636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:50.523938 22636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:50.524295 22636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:50.542853 22636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46337
I20260621 02:43:50.542896 22702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46337 every 8 connection(s)
I20260621 02:43:50.544328 22636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:50.548418 22703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:50.549723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 22636
I20260621 02:43:50.549993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:50.550599 388 external_mini_cluster.cc:1428] Setting key d941cf80381c8a0cee7762c5a8cca7f2
I20260621 02:43:50.558758 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:50.552393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41667 (local address 127.0.97.62:46337)
0621 02:43:50.553179 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:50.553192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:50.553231 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:43:50.555496 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:50.555518 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:50.555842 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:50.556231 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:50.557209 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:50.557215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:50.557228 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:50.557241 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:50.557365 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:50.557398 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:50.557617 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:50.557742 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 02:43:50.557920 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":354,"threads_started":1}
I20260621 02:43:50.562932 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:50.563050 22703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532: Bootstrap starting.
I20260621 02:43:50.565984 22703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:50.566972 22703 log.cc:824] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:50.569867 22703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532: No bootstrap required, opened a new log
I20260621 02:43:50.570035 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:50.567214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39905 (local address 127.0.97.62:46337)
0621 02:43:50.567426 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:50.567436 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:50.567930 (+ 494us) server_negotiation.cc:407] Connection header received
0621 02:43:50.568124 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:50.568129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:50.568186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:50.568291 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:50.569227 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:50.569233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:50.569238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:43:50.569255 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:50.569345 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:43:50.569364 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:50.569495 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:50.569623 (+ 128us) server_negotiation.cc:299] Negotiation successful
0621 02:43:50.569682 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:43:50.577916 22703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbd9e59d804ac597be24c8a5c94532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46337 } }
I20260621 02:43:50.578305 22703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:43:50.578414 22703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecbd9e59d804ac597be24c8a5c94532, State: Initialized, Role: FOLLOWER
I20260621 02:43:50.578987 22703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbd9e59d804ac597be24c8a5c94532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46337 } }
I20260621 02:43:50.579216 22703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:43:50.579367 22703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:43:50.579552 22703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:50.581822 22703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbd9e59d804ac597be24c8a5c94532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46337 } }
I20260621 02:43:50.582310 22703 leader_election.cc:302] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ecbd9e59d804ac597be24c8a5c94532; no voters:
I20260621 02:43:50.582913 22703 leader_election.cc:288] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:43:50.582981 22713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:50.584349 22713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [term 1 LEADER]: Becoming Leader. State: Replica: 7ecbd9e59d804ac597be24c8a5c94532, State: Running, Role: LEADER
I20260621 02:43:50.584849 22713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbd9e59d804ac597be24c8a5c94532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46337 } }
I20260621 02:43:50.585187 22703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:50.587491 22715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ecbd9e59d804ac597be24c8a5c94532. Latest consensus state: current_term: 1 leader_uuid: "7ecbd9e59d804ac597be24c8a5c94532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbd9e59d804ac597be24c8a5c94532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46337 } } }
I20260621 02:43:50.587886 22715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:50.588415 22714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ecbd9e59d804ac597be24c8a5c94532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbd9e59d804ac597be24c8a5c94532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46337 } } }
I20260621 02:43:50.588740 22714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:50.591187 22721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:50.597077 22721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:50.606472 22721 catalog_manager.cc:1347] Generated new cluster ID: 04c0ac0448954b6494145e262e7fef04
I20260621 02:43:50.606611 22721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:50.620963 22721 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:50.621517 22721 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:50.632572 22721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ecbd9e59d804ac597be24c8a5c94532: Generated new TSK 0
I20260621 02:43:50.633471 22721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:43:50.644611 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22636
W20260621 02:43:50.646231 22698 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41667
W20260621 02:43:50.646893 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22636: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:43:50.647028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 22636
2026-06-21T02:43:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1866 ms)
[----------] 18 tests from AuthTokenIssuingTest (34398 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:50 dist-test-slave-jkn7 krb5kdc[22735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:50 dist-test-slave-jkn7 krb5kdc[22735](info): set up 2 sockets
Jun 21 02:43:50 dist-test-slave-jkn7 krb5kdc[22735](info): commencing operation
krb5kdc: starting...
W20260621 02:43:51.753876 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:51 dist-test-slave-jkn7 krb5kdc[22735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:51.994616 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37671
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37671,127.0.97.61:37957,127.0.97.60:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:52.308296 22748 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:43:52.308606 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:52.308670 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:52.308755 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:52.315920 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:52.316009 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:52.316066 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:52.316109 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:52.316171 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:52.325759 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37671,127.0.97.61:37957,127.0.97.60:32999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:52.326996 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:52.328226 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:52.340015 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:52.340080 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:52.340081 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:52.342572 22748 server_base.cc:1034] running on GCE node
I20260621 02:43:52.343287 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:52.344597 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:52.345819 22748 hybrid_clock.cc:648] HybridClock initialized: now 1782009832345759 us; error 24 us; skew 500 ppm
I20260621 02:43:52.346153 22748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:52 dist-test-slave-jkn7 krb5kdc[22735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:52.349370 22748 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:52.350807 22748 webserver.cc:460] Webserver started at http://127.0.97.62:40295/ using document root <none> and password file <none>
I20260621 02:43:52.351502 22748 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:52.351631 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:52.352022 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:52.354918 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d"
format_stamp: "Formatted at 2026-06-21 02:43:52 on dist-test-slave-jkn7"
server_key: "ee8c12be973796f724b083d72e2bc16f"
server_key_iv: "7fd9dc2aa15e5a825ab10ca651839b74"
server_key_version: "encryptionkey@0"
I20260621 02:43:52.355675 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d"
format_stamp: "Formatted at 2026-06-21 02:43:52 on dist-test-slave-jkn7"
server_key: "ee8c12be973796f724b083d72e2bc16f"
server_key_iv: "7fd9dc2aa15e5a825ab10ca651839b74"
server_key_version: "encryptionkey@0"
I20260621 02:43:52.360649 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260621 02:43:52.363925 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:52.365192 22748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:43:52.365408 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d"
format_stamp: "Formatted at 2026-06-21 02:43:52 on dist-test-slave-jkn7"
server_key: "ee8c12be973796f724b083d72e2bc16f"
server_key_iv: "7fd9dc2aa15e5a825ab10ca651839b74"
server_key_version: "encryptionkey@0"
I20260621 02:43:52.365593 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:52.394882 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:52.399106 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:52.399384 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:52.417557 22748 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37671
I20260621 02:43:52.417574 22814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37671 every 8 connection(s)
I20260621 02:43:52.418810 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:52.422176 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:52.425339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 22748
I20260621 02:43:52.425545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:52.426344 388 external_mini_cluster.cc:1428] Setting key c4a63894bd1dbcdd0e9aa9fd0401eb45
I20260621 02:43:52.430449 22815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:52 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260621 02:43:52.454424 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.434069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37671 (local address 127.0.0.1:32850)
0621 02:43:52.434969 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.435015 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:52.437332 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:52.438141 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:52.438151 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:52.438924 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:52.439388 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:52.439414 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.440917 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.440925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:52.442806 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:52.442823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.443712 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.443721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:52.443930 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:52.444661 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:52.444694 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:52.447296 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:52.450423 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:52.450433 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:52.450444 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:52.450814 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:52.451202 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:52.451209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:52.451212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:52.451374 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:52.452113 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:52.452125 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:52.452334 (+ 209us) client_negotiation.cc:769] Sending connection context
0621 02:43:52.452566 (+ 232us) client_negotiation.cc:240] Negotiation successful
0621 02:43:52.453161 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":166,"thread_start_us":390,"threads_started":1}
I20260621 02:43:52.454461 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.434069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.0.97.62:37671)
0621 02:43:52.435952 (+ 1883us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:52.435961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:52.436001 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:43:52.437686 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:52.437692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:52.437914 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:52.438064 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:52.439609 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.440756 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.442953 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.443470 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.443607 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:52.447521 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:52.447528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:52.447539 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:52.447566 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:52.450285 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:52.450952 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:52.450958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:52.450961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:52.451089 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:52.451488 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:52.451494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:52.451502 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:52.451984 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:52.452187 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:52.452680 (+ 493us) server_negotiation.cc:299] Negotiation successful
0621 02:43:52.453060 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":1690,"spinlock_wait_cycles":16768,"thread_start_us":1455,"threads_started":1}
I20260621 02:43:52.457185 22815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:43:52.458207 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.457848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60322)
0621 02:43:52.458001 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.458098 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260621 02:43:52.458513 22766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
I20260621 02:43:52.460004 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.459601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60334)
0621 02:43:52.459808 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.459894 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260621 02:43:52.460300 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260621 02:43:52.480168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37957
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37671,127.0.97.61:37957,127.0.97.60:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:43:52.509613 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } attempt: 1
I20260621 02:43:52.510809 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.510293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60338)
0621 02:43:52.510535 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.510637 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 02:43:52.511821 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.511439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60344)
0621 02:43:52.511593 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.511673 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:43:52.512027 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
I20260621 02:43:52.580308 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } attempt: 2
I20260621 02:43:52.581676 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.581054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60346)
0621 02:43:52.581371 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.581489 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260621 02:43:52.582973 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.582520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60356)
0621 02:43:52.582688 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.582809 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 02:43:52.583217 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
I20260621 02:43:52.674489 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } attempt: 3
I20260621 02:43:52.675715 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.675198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60364)
0621 02:43:52.675428 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.675541 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 02:43:52.676846 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.676442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60374)
0621 02:43:52.676600 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.676685 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260621 02:43:52.677084 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
W20260621 02:43:52.806453 22822 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:43:52.806826 22822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:52.806958 22822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:52.807116 22822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:52.814764 22822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:52.814857 22822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:52.814949 22822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:52.815030 22822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:52.815143 22822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:52.820339 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } attempt: 4
I20260621 02:43:52.821621 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.821003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60384)
0621 02:43:52.821369 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.821472 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260621 02:43:52.822674 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.822306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60396)
0621 02:43:52.822451 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.822543 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:43:52.822898 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37957: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37957: connect: Connection refused (error 111)
I20260621 02:43:52.825857 22822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37671,127.0.97.61:37957,127.0.97.60:32999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:52.827060 22822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:52.828255 22822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:52.839196 22828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:52.839236 22830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:52.839383 22827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:52.840209 22822 server_base.cc:1034] running on GCE node
I20260621 02:43:52.840910 22822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:52.842113 22822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:52.843318 22822 hybrid_clock.cc:648] HybridClock initialized: now 1782009832843254 us; error 37 us; skew 500 ppm
I20260621 02:43:52.843573 22822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:52 dist-test-slave-jkn7 krb5kdc[22735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:52.846426 22822 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260621 02:43:52.847640 22822 webserver.cc:460] Webserver started at http://127.0.97.61:46331/ using document root <none> and password file <none>
I20260621 02:43:52.848224 22822 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:52.848305 22822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:52.848536 22822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:52.851253 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/data/instance:
uuid: "58738a421931486aaa6878852f80c5f2"
format_stamp: "Formatted at 2026-06-21 02:43:52 on dist-test-slave-jkn7"
server_key: "b6ae91788dbd315836a6d07f7484262f"
server_key_iv: "8bb88b19ab52ebe95cc0e85d47a61dd5"
server_key_version: "encryptionkey@0"
I20260621 02:43:52.852008 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/wal/instance:
uuid: "58738a421931486aaa6878852f80c5f2"
format_stamp: "Formatted at 2026-06-21 02:43:52 on dist-test-slave-jkn7"
server_key: "b6ae91788dbd315836a6d07f7484262f"
server_key_iv: "8bb88b19ab52ebe95cc0e85d47a61dd5"
server_key_version: "encryptionkey@0"
I20260621 02:43:52.856662 22822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:43:52.859757 22837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:52.861037 22822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:43:52.861161 22822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/wal
uuid: "58738a421931486aaa6878852f80c5f2"
format_stamp: "Formatted at 2026-06-21 02:43:52 on dist-test-slave-jkn7"
server_key: "b6ae91788dbd315836a6d07f7484262f"
server_key_iv: "8bb88b19ab52ebe95cc0e85d47a61dd5"
server_key_version: "encryptionkey@0"
I20260621 02:43:52.861399 22822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:52.877902 22822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:52.881839 22822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:52.882028 22822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:52.898855 22822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37957
I20260621 02:43:52.898871 22888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37957 every 8 connection(s)
I20260621 02:43:52.900043 22822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
I20260621 02:43:52.902724 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 22822
I20260621 02:43:52.902904 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-1/wal/instance
I20260621 02:43:52.903337 388 external_mini_cluster.cc:1428] Setting key 9c84bb52a7971b721c8cfa555eae0c05
I20260621 02:43:52.903543 22889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:52.911085 22889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:52 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260621 02:43:52.932294 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.914177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.0.97.62:37671)
0621 02:43:52.914412 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:52.914421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:52.915537 (+ 1116us) server_negotiation.cc:407] Connection header received
0621 02:43:52.917449 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:52.917465 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:52.917559 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:52.917676 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:52.919064 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.920199 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.921757 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.922546 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.922690 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:52.926214 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:52.926223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:52.926230 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:52.926279 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:52.929115 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:52.930045 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:52.930049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:52.930051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:52.930167 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:52.930652 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:52.930655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:52.930657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:52.930920 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:52.931036 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:52.931911 (+ 875us) server_negotiation.cc:299] Negotiation successful
0621 02:43:52.932074 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:43:52.933789 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.914572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37671 (local address 127.0.0.1:32856)
0621 02:43:52.915359 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.915402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:52.917115 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:52.917866 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:52.917873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:52.918353 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:52.918881 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:52.918898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.920355 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.920360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:52.921613 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:52.921625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.922796 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.922801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:52.923064 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:52.923806 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:52.923828 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:52.925973 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:52.929245 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:52.929252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:52.929266 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:52.929909 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:52.930280 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:52.930287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:52.930293 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:52.930535 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:52.931051 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:52.931061 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:52.931537 (+ 476us) client_negotiation.cc:769] Sending connection context
0621 02:43:52.931828 (+ 291us) client_negotiation.cc:240] Negotiation successful
0621 02:43:52.932377 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":340,"threads_started":1}
I20260621 02:43:52.935389 22889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:52 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260621 02:43:52.950055 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.936148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60402)
0621 02:43:52.936364 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.936383 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:52.936480 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:52.937644 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:52.937652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:52.938166 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:52.938384 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:52.938398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.940026 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.940030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:52.940846 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:52.940857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.941702 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.941708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:52.941896 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:52.942474 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:52.942492 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:52.944419 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:52.947587 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:52.947593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:52.947597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:52.947957 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:52.948327 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:52.948331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:52.948335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:52.948463 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:52.949047 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:52.949051 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:52.949199 (+ 148us) client_negotiation.cc:769] Sending connection context
0621 02:43:52.949388 (+ 189us) client_negotiation.cc:240] Negotiation successful
0621 02:43:52.949668 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 02:43:52.950172 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.936468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.0.97.61:37957)
0621 02:43:52.937123 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:52.937132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:52.937160 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:43:52.937207 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:52.937217 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:52.937478 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:52.937605 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:52.938558 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.939886 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.940985 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:52.941563 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:52.941694 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:52.944624 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:52.944634 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:52.944650 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:52.944680 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:52.947440 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:52.948088 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:52.948098 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:52.948106 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:52.948214 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:52.948562 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:52.948570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:52.948573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:52.948912 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:52.949123 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:52.949531 (+ 408us) server_negotiation.cc:299] Negotiation successful
0621 02:43:52.949780 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":487,"thread_start_us":344,"threads_started":1}
I20260621 02:43:52.952363 22889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:43:52.953568 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.953112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:41988)
0621 02:43:52.953273 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.953418 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 02:43:52.953855 22841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
I20260621 02:43:52.955341 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:52.955025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42004)
0621 02:43:52.955165 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:52.955245 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:43:52.955564 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260621 02:43:52.960412 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32999
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37671,127.0.97.61:37957,127.0.97.60:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:43:53.004853 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } attempt: 1
I20260621 02:43:53.006006 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.005582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42006)
0621 02:43:53.005766 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.005851 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 02:43:53.006971 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.006634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42016)
0621 02:43:53.006791 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.006872 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 02:43:53.007162 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
I20260621 02:43:53.075376 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } attempt: 2
I20260621 02:43:53.076442 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.075993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42026)
0621 02:43:53.076187 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.076291 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 02:43:53.077505 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.077136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42042)
0621 02:43:53.077340 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.077400 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:43:53.077700 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
I20260621 02:43:53.122174 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } attempt: 5
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260621 02:43:53.136814 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.122870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60404)
0621 02:43:53.123154 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.123180 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:53.123350 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:53.123748 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:53.123755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:53.124361 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:53.124673 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.124691 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.125848 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.125854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.127256 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.127271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.128042 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.128048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.128261 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.128971 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:53.128994 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:53.131360 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:53.134446 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.134452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.134457 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.134825 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.135314 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.135317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.135320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.135447 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.136083 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:53.136087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:53.136225 (+ 138us) client_negotiation.cc:769] Sending connection context
0621 02:43:53.136367 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 02:43:53.136560 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 02:43:53.137069 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.123046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.0.97.61:37957)
0621 02:43:53.123203 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.123207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.123264 (+ 57us) server_negotiation.cc:407] Connection header received
0621 02:43:53.123506 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.123510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.123572 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.123654 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.124882 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.125685 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.127432 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.127888 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.127987 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.131559 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.131564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.131568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.131599 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.134307 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.135000 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.135007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.135012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.135183 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.135567 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.135574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.135579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.135951 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.136104 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.136538 (+ 434us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.136771 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 02:43:53.138191 22815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:43:53.139349 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.138962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42054)
0621 02:43:53.139137 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.139216 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 02:43:53.140610 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.140293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42064)
0621 02:43:53.140434 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.140516 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:43:53.140816 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
I20260621 02:43:53.168954 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } attempt: 3
I20260621 02:43:53.170154 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.169652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42078)
0621 02:43:53.169865 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.169953 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260621 02:43:53.171264 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.170871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42090)
0621 02:43:53.171087 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.171157 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260621 02:43:53.171450 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
I20260621 02:43:53.192049 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } attempt: 1
I20260621 02:43:53.193073 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.192634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42104)
0621 02:43:53.192831 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.192917 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 02:43:53.194103 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.193801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42112)
0621 02:43:53.193947 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.194007 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:43:53.194303 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
I20260621 02:43:53.262499 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } attempt: 2
I20260621 02:43:53.263607 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.263132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42124)
0621 02:43:53.263351 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.263439 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 02:43:53.264632 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.264300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42138)
0621 02:43:53.264435 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.264506 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 02:43:53.264817 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
W20260621 02:43:53.279058 22896 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:43:53.279393 22896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:53.279517 22896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:53.279649 22896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:53.286656 22896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:53.286767 22896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:53.286897 22896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:53.286989 22896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:53.287075 22896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:53.296412 22896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37671,127.0.97.61:37957,127.0.97.60:32999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:53.297667 22896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:53.298940 22896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:53.310364 22901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:53.310328 22904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:53.310379 22902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:53.311230 22896 server_base.cc:1034] running on GCE node
I20260621 02:43:53.311986 22896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:53.313357 22896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:53.314630 22896 hybrid_clock.cc:648] HybridClock initialized: now 1782009833314538 us; error 40 us; skew 500 ppm
I20260621 02:43:53.314648 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } attempt: 4
I20260621 02:43:53.315016 22896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:43:53.315794 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.315248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42148)
0621 02:43:53.315497 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.315595 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 02:43:53.316772 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.316382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42150)
0621 02:43:53.316559 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.316645 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 02:43:53.316970 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:53.318689 22896 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260621 02:43:53.320108 22896 webserver.cc:460] Webserver started at http://127.0.97.60:44871/ using document root <none> and password file <none>
I20260621 02:43:53.320794 22896 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:53.320909 22896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:53.321235 22896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:53.324112 22896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/data/instance:
uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00"
format_stamp: "Formatted at 2026-06-21 02:43:53 on dist-test-slave-jkn7"
server_key: "4fc987d59f3afc2d01eca75b7fc43f46"
server_key_iv: "5e4820d1ad73f9e3d930b5d7c4952474"
server_key_version: "encryptionkey@0"
I20260621 02:43:53.324892 22896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/wal/instance:
uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00"
format_stamp: "Formatted at 2026-06-21 02:43:53 on dist-test-slave-jkn7"
server_key: "4fc987d59f3afc2d01eca75b7fc43f46"
server_key_iv: "5e4820d1ad73f9e3d930b5d7c4952474"
server_key_version: "encryptionkey@0"
I20260621 02:43:53.329882 22896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:43:53.333163 22911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:53.334542 22896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:43:53.334714 22896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/wal
uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00"
format_stamp: "Formatted at 2026-06-21 02:43:53 on dist-test-slave-jkn7"
server_key: "4fc987d59f3afc2d01eca75b7fc43f46"
server_key_iv: "5e4820d1ad73f9e3d930b5d7c4952474"
server_key_version: "encryptionkey@0"
I20260621 02:43:53.334929 22896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:53.349026 22896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:53.353130 22896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:53.353492 22896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:53.371057 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } attempt: 3
I20260621 02:43:53.372323 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.371719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42160)
0621 02:43:53.372012 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.372130 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 02:43:53.373375 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.372928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42168)
0621 02:43:53.373125 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.373194 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260621 02:43:53.373600 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32999: connect: Connection refused (error 111)
I20260621 02:43:53.377848 22896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:32999
I20260621 02:43:53.377877 22962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:32999 every 8 connection(s)
I20260621 02:43:53.379129 22896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
I20260621 02:43:53.382716 22963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:53.382899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 22896
I20260621 02:43:53.383081 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009483336832-388-0/minicluster-data/master-2/wal/instance
I20260621 02:43:53.383571 388 external_mini_cluster.cc:1428] Setting key 65e3adffb510d6072bc68d7155ee156c
I20260621 02:43:53.391552 22963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:53.406531 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.385361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.0.97.60:32999)
0621 02:43:53.386262 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.386273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.386313 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:43:53.388580 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.388601 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.388969 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.389278 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.390970 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.392203 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.393507 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.394350 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.394571 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.398971 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.398979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.398989 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.399023 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.402228 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.402921 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.402927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.402933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.403067 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.403450 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.403453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.403455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.404074 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.404297 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.404855 (+ 558us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.405277 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":411,"threads_started":1}
I20260621 02:43:53.408792 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.393827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.0.97.62:37671)
0621 02:43:53.394031 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.394038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.394631 (+ 593us) server_negotiation.cc:407] Connection header received
0621 02:43:53.394907 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.394914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.395005 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.395108 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.396208 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.397343 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.398653 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.399084 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.399216 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.402813 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.402821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.402828 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.402873 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.405950 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.406811 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.406817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.406823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.406983 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.407373 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.407378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.407382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.407721 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.407893 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.408389 (+ 496us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.408551 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:43:53.409039 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.393778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37671 (local address 127.0.0.1:32858)
0621 02:43:53.394486 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.394535 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:53.394755 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:53.395183 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:53.395191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:53.395824 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:53.396050 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.396058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.397488 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.397492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.398512 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.398524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.399286 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.399304 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.399534 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.400148 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:53.400174 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:53.402624 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:53.406074 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.406084 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.406090 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.406656 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.407113 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.407117 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.407119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.407264 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.407897 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:53.407904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:53.408089 (+ 185us) client_negotiation.cc:769] Sending connection context
0621 02:43:53.408291 (+ 202us) client_negotiation.cc:240] Negotiation successful
0621 02:43:53.408599 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":339,"threads_started":1}
I20260621 02:43:53.410436 22963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:53.424201 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.411192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37957 (local address 127.0.0.1:60408)
0621 02:43:53.411354 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.411375 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:53.411459 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:53.411795 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:53.411798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:53.412100 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:53.412225 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.412232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.413221 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.413226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.414124 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.414136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.414827 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.414833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.415005 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.415624 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:53.415641 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:53.417970 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:53.421694 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.421703 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.421708 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.422160 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.422542 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.422546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.422550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.422718 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.423304 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:53.423308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:53.423476 (+ 168us) client_negotiation.cc:769] Sending connection context
0621 02:43:53.423614 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 02:43:53.423881 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 02:43:53.424635 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.409833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.0.97.62:37671)
0621 02:43:53.410378 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.410382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.410396 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:43:53.410432 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.410435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.410485 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.410557 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.412157 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.412889 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.414169 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.414608 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.414751 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.418759 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.418765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.418769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.418814 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.421864 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.422614 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.422622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.422627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.422771 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.423146 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.423151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.423155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.423504 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.423792 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.424107 (+ 315us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.424311 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455}
I20260621 02:43:53.425056 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.411243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.0.97.61:37957)
0621 02:43:53.411404 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.411408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.411424 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:43:53.411578 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.411585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.411665 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.411776 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.412389 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.413072 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.414276 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.414677 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.414820 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.418166 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.418173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.418178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.418217 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.421557 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.422275 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.422282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.422286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.422415 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.422845 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.422851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.422855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.423180 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.423321 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.424488 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.424713 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 02:43:53.426036 22963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260621 02:43:53.440240 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.426767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.0.97.60:32999)
0621 02:43:53.426924 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.426929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.427142 (+ 213us) server_negotiation.cc:407] Connection header received
0621 02:43:53.427279 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.427283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.427349 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.427441 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.428300 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.429017 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.430318 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.430700 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.430804 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.434179 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.434185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.434190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.434228 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.437534 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.438406 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.438412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.438415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.438553 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.438911 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.438916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.438919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.439298 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.439446 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.439781 (+ 335us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.440006 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:43:53.440287 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.426833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42188)
0621 02:43:53.427048 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.427067 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:53.427170 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:53.427489 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:53.427493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:53.427960 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:53.428145 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.428157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.429142 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.429146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.430174 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.430187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.430866 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.430873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.431086 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.431671 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:53.431687 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:53.434012 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:53.437681 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.437684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.437687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.438283 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.438661 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.438664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.438667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.438800 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.439429 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:53.439433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:53.439554 (+ 121us) client_negotiation.cc:769] Sending connection context
0621 02:43:53.439665 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 02:43:53.439850 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260621 02:43:53.442458 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.426882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.0.97.61:37957)
0621 02:43:53.427041 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.427046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.427373 (+ 327us) server_negotiation.cc:407] Connection header received
0621 02:43:53.427437 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.427442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.427505 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.427590 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.428965 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.429808 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.431289 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.431662 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.431753 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.436258 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.436264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.436269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.436328 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.439826 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.440620 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.440625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.440628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.440816 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.441180 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.441188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.441193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.441550 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.441748 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.442065 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.442215 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:43:53.445775 22963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00: Bootstrap starting.
I20260621 02:43:53.446975 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:53.448942 22963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:53.450255 22963 log.cc:824] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:53.453202 22963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00: No bootstrap required, opened a new log
I20260621 02:43:53.449527 22790 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32864
I20260621 02:43:53.459962 22963 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } }
I20260621 02:43:53.460263 22963 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1bde81ca4b4ecea55fd48dbb7acb00, State: Initialized, Role: FOLLOWER
I20260621 02:43:53.460677 22963 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } }
I20260621 02:43:53.461527 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.461946 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:53.462707 22963 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:43:53.471724 22980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:43:53.471864 22980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:43:53.470009 22896 master_runner.cc:186] Error getting master registration for 127.0.97.62:37671: OK, instance_id { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" instance_seqno: 1782009832412522 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:43:53.481025 22896 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:43:53.550812 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } attempt: 4
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260621 02:43:53.567582 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.551560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42196)
0621 02:43:53.551787 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.551810 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:53.551925 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:53.552402 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:53.552408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:53.552915 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:53.553131 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.553144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.554593 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.554599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.556095 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.556121 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.557103 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.557109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.557374 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.558106 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:53.558131 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:53.560826 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:53.564755 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.564761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.564765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.565272 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.565726 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.565732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.565736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.565903 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.566545 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:53.566550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:53.566725 (+ 175us) client_negotiation.cc:769] Sending connection context
0621 02:43:53.566871 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 02:43:53.567175 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 02:43:53.567613 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.551607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.0.97.60:32999)
0621 02:43:53.551829 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.551833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.551870 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:43:53.552120 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.552132 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.552228 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.552347 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.553310 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.554441 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.556247 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.556955 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.557071 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.561016 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.561022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.561027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.561066 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.564609 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.565405 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.565411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.565414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.565570 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.566027 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.566033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.566037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.566423 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.566583 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.567004 (+ 421us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.567221 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260621 02:43:53.572358 22815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d: Bootstrap starting.
I20260621 02:43:53.576234 22815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:53.577267 22815 log.cc:824] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:53.579912 22815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d: No bootstrap required, opened a new log
I20260621 02:43:53.584861 22815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } }
I20260621 02:43:53.585172 22815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc5afd5670c4e97ab30a9d4ee5ae31d, State: Initialized, Role: FOLLOWER
I20260621 02:43:53.585738 22815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } }
I20260621 02:43:53.586687 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.587033 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:53.587756 22815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:53.587863 22790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 02:43:53.588063 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:53.591455 22790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } }
I20260621 02:43:53.593426 22790 leader_election.cc:288] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [CANDIDATE]: Term 1 election: Requested vote from peers 58738a421931486aaa6878852f80c5f2 (127.0.97.61:37957), fc1bde81ca4b4ecea55fd48dbb7acb00 (127.0.97.60:32999)
I20260621 02:43:53.593961 22938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00"
I20260621 02:43:53.594486 22938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:53.598284 22938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fc5afd5670c4e97ab30a9d4ee5ae31d in term 1.
I20260621 02:43:53.598994 22767 leader_election.cc:302] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fc5afd5670c4e97ab30a9d4ee5ae31d, fc1bde81ca4b4ecea55fd48dbb7acb00; no voters:
I20260621 02:43:53.599475 22983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [term 1 FOLLOWER]: Leader election won for term 1
W20260621 02:43:53.610159 22917 tablet.cc:2369] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:43:53.611591 22995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:43:53.611759 22995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:43:53.613418 22983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [term 1 LEADER]: Becoming Leader. State: Replica: 9fc5afd5670c4e97ab30a9d4ee5ae31d, State: Running, Role: LEADER
I20260621 02:43:53.614060 22983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } }
I20260621 02:43:53.616158 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } attempt: 5
I20260621 02:43:53.601318 22748 master_runner.cc:186] Error getting master registration for 127.0.97.61:37957: OK, instance_id { permanent_uuid: "58738a421931486aaa6878852f80c5f2" instance_seqno: 1782009832893935 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Jun 21 02:43:53 dist-test-slave-jkn7 krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260621 02:43:53.626629 22748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:43:53.634318 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.616765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32999 (local address 127.0.0.1:42200)
0621 02:43:53.617168 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:53.617189 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:53.617348 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:53.618383 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:53.618389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:53.618914 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:53.619122 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.619134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.620668 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.620674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.622158 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:53.622173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.623347 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.623353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:53.623559 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.624247 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:53.624278 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:53.627055 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:53.631493 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.631498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.631503 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.632030 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.632487 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:53.632492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:53.632496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:53.632660 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:53.633343 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:53.633348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:53.633504 (+ 156us) client_negotiation.cc:769] Sending connection context
0621 02:43:53.633640 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 02:43:53.633929 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260621 02:43:53.635298 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.617691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.0.97.60:32999)
0621 02:43:53.617870 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.617876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.617895 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:43:53.617959 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.617965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.618045 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.618143 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.619301 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.620470 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.622354 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.623011 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.623130 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.627270 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.627277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.627283 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.627320 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.631311 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.632174 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.632180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.632183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.632343 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.632780 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.632788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.632797 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.633116 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.633258 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.634626 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.634874 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:43:53.636559 22997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc5afd5670c4e97ab30a9d4ee5ae31d. Latest consensus state: current_term: 1 leader_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.637037 22997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:53.642381 22889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2: Bootstrap starting.
I20260621 02:43:53.647061 22889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:53.648607 22889 log.cc:824] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2: Log is configured to *not* fsync() on all Append() calls
W20260621 02:43:53.652061 22769 tablet.cc:2369] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:43:53.665225 23003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:53.667512 23003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:53.675372 22889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2: No bootstrap required, opened a new log
I20260621 02:43:53.682271 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.637137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.0.97.62:37671)
0621 02:43:53.637367 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.637373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.649428 (+ 12055us) server_negotiation.cc:407] Connection header received
0621 02:43:53.653115 (+ 3687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.653125 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.653225 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.653328 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.656026 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.657393 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.662195 (+ 4802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.662974 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.663082 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.667659 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.667666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.667671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.667711 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.671682 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.679753 (+ 8071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.679759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.679763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.679917 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.680354 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.680360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.680364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.680712 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.681012 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.681611 (+ 599us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.681839 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:43:53.688745 22889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } }
I20260621 02:43:53.689463 22889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58738a421931486aaa6878852f80c5f2, State: Initialized, Role: FOLLOWER
I20260621 02:43:53.690268 22889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } }
I20260621 02:43:53.701218 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "58738a421931486aaa6878852f80c5f2"
I20260621 02:43:53.701694 22864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:53.705044 22864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fc5afd5670c4e97ab30a9d4ee5ae31d in term 1.
I20260621 02:43:53.705968 22889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:53.707471 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.707881 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:53.708211 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.645223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.0.97.61:37957)
0621 02:43:53.645405 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.645410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.661466 (+ 16056us) server_negotiation.cc:407] Connection header received
0621 02:43:53.662158 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.662164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.662245 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.662313 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.677964 (+ 15651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.679064 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.682251 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.682952 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.683050 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.691802 (+ 8752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.691811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.691815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.691860 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.695692 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.703817 (+ 8125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.703825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.703829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.704011 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.704611 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.704616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.704620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.705002 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.705253 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.707561 (+ 2308us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.707835 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:43:53.710911 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:53.645151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42206 (local address 127.0.97.60:32999)
0621 02:43:53.645379 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:53.645385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:53.663281 (+ 17896us) server_negotiation.cc:407] Connection header received
0621 02:43:53.663819 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:53.663826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:53.663905 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:53.663975 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:53.677905 (+ 13930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.679021 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.686235 (+ 7214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:53.686937 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:53.687045 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:53.696493 (+ 9448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:53.696501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:53.696505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:53.696548 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:53.700413 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.705753 (+ 5340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.705759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.705763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.705910 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:53.706380 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:53.706386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:53.706390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:53.706746 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:53.706983 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:53.710251 (+ 3268us) server_negotiation.cc:299] Negotiation successful
0621 02:43:53.710524 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 02:43:53.715462 22864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [term 1 FOLLOWER]: Refusing update from remote peer 9fc5afd5670c4e97ab30a9d4ee5ae31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:53.715503 22938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [term 1 FOLLOWER]: Refusing update from remote peer 9fc5afd5670c4e97ab30a9d4ee5ae31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:53.716368 22997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:53.716658 22983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:53.752177 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc5afd5670c4e97ab30a9d4ee5ae31d. Latest consensus state: current_term: 1 leader_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.752542 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:53.754455 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc5afd5670c4e97ab30a9d4ee5ae31d. Latest consensus state: current_term: 1 leader_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.754745 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:53.755828 23007 mvcc.cc:205] Tried to move back new op lower bound from 7299112278845652992 to 7299112278488539136. Current Snapshot: MvccSnapshot[applied={T|T < 7299112278845652992}]
I20260621 02:43:53.757755 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.758059 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:53.762641 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.762897 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:53.763360 23003 catalog_manager.cc:1347] Generated new cluster ID: fafa69c647e846ae83037941e864b9c6
I20260621 02:43:53.763473 23003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:43:53.763958 22997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.764221 22997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:53.769515 23012 mvcc.cc:205] Tried to move back new op lower bound from 7299112278845652992 to 7299112278488539136. Current Snapshot: MvccSnapshot[applied={T|T < 7299112278845652992}]
I20260621 02:43:53.774520 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fc5afd5670c4e97ab30a9d4ee5ae31d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37671 } } peers { permanent_uuid: "58738a421931486aaa6878852f80c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37957 } } peers { permanent_uuid: "fc1bde81ca4b4ecea55fd48dbb7acb00" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32999 } } }
I20260621 02:43:53.774814 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:53.833984 23003 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:53.841051 23026 catalog_manager.cc:1259] Loaded cluster ID: fafa69c647e846ae83037941e864b9c6
I20260621 02:43:53.841152 23026 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2: loading cluster ID for follower catalog manager: success
I20260621 02:43:53.844250 23003 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:53.844770 23026 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2: acquiring CA information for follower catalog manager: success
W20260621 02:43:53.850193 23026 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260621 02:43:53.876066 23003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fc5afd5670c4e97ab30a9d4ee5ae31d: Generated new TSK 0
I20260621 02:43:53.877341 23003 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:43:54.035758 22818 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.009401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.0.97.62:37671)
0621 02:43:54.009635 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.009645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.032402 (+ 22757us) server_negotiation.cc:407] Connection header received
0621 02:43:54.034138 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.034145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.034242 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.034324 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.035522 (+ 1198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32872: BlockingRecv error: recv got EOF from 127.0.0.1:32872 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260621 02:43:54.040916 22966 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.027067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.0.97.60:32999)
0621 02:43:54.027259 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.027265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.037816 (+ 10551us) server_negotiation.cc:407] Connection header received
0621 02:43:54.039553 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.039559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.039642 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.039709 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.040658 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42222: BlockingRecv error: recv got EOF from 127.0.0.1:42222 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 02:43:54.042001 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.015509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.97.61:37957)
0621 02:43:54.015719 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.015728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.037133 (+ 21405us) server_negotiation.cc:407] Connection header received
0621 02:43:54.038481 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.038488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.038595 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.038677 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.041769 (+ 3092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60430: BlockingRecv error: recv got EOF from 127.0.0.1:60430 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260621 02:43:54.053592 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.051123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.0.97.61:37957)
0621 02:43:54.051307 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.051314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.051753 (+ 439us) server_negotiation.cc:407] Connection header received
0621 02:43:54.052061 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.052066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.052239 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.052320 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.053306 (+ 986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60440: BlockingRecv error: recv got EOF from 127.0.0.1:60440 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260621 02:43:54.061178 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.059098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.0.97.61:37957)
0621 02:43:54.059248 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.059254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.059684 (+ 430us) server_negotiation.cc:407] Connection header received
0621 02:43:54.059929 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.059935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.060016 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.060084 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.060967 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60448: BlockingRecv error: recv got EOF from 127.0.0.1:60448 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260621 02:43:54.072335 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.070130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.97.61:37957)
0621 02:43:54.070350 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.070355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.070803 (+ 448us) server_negotiation.cc:407] Connection header received
0621 02:43:54.071062 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.071067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.071172 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.071241 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.072102 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60458: BlockingRecv error: recv got EOF from 127.0.0.1:60458 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260621 02:43:54.082311 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.080204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.0.97.61:37957)
0621 02:43:54.080422 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.080430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.080912 (+ 482us) server_negotiation.cc:407] Connection header received
0621 02:43:54.081124 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.081130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.081211 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.081338 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.082072 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60472: BlockingRecv error: recv got EOF from 127.0.0.1:60472 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260621 02:43:54.095933 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.093694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.0.97.61:37957)
0621 02:43:54.093872 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.093879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.094432 (+ 553us) server_negotiation.cc:407] Connection header received
0621 02:43:54.094639 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.094645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.094744 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.094849 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.095681 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60478: BlockingRecv error: recv got EOF from 127.0.0.1:60478 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 02:43:54.129704 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.127273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.0.97.61:37957)
0621 02:43:54.127499 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.127506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.128078 (+ 572us) server_negotiation.cc:407] Connection header received
0621 02:43:54.128296 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.128303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.128401 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.128521 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.129427 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60490: BlockingRecv error: recv got EOF from 127.0.0.1:60490 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 02:43:54.168416 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.166047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.0.97.61:37957)
0621 02:43:54.166261 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.166268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.166817 (+ 549us) server_negotiation.cc:407] Connection header received
0621 02:43:54.167076 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.167082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.167188 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.167298 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.168143 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60492: BlockingRecv error: recv got EOF from 127.0.0.1:60492 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260621 02:43:54.240936 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.238829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.0.97.61:37957)
0621 02:43:54.239083 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.239090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.239476 (+ 386us) server_negotiation.cc:407] Connection header received
0621 02:43:54.239718 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.239729 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.239803 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.239922 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.240673 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60496: BlockingRecv error: recv got EOF from 127.0.0.1:60496 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260621 02:43:54.374127 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.372191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.97.61:37957)
0621 02:43:54.372383 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.372391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.372702 (+ 311us) server_negotiation.cc:407] Connection header received
0621 02:43:54.372942 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.372946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.373016 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.373123 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.373847 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60508: BlockingRecv error: recv got EOF from 127.0.0.1:60508 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:43:54.473809 22980 catalog_manager.cc:1259] Loaded cluster ID: fafa69c647e846ae83037941e864b9c6
I20260621 02:43:54.473912 22980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00: loading cluster ID for follower catalog manager: success
I20260621 02:43:54.476308 22980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00: acquiring CA information for follower catalog manager: success
I20260621 02:43:54.478066 22980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc1bde81ca4b4ecea55fd48dbb7acb00: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:43:54.636730 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:54.634772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.0.97.61:37957)
0621 02:43:54.634987 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:54.634995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:54.635354 (+ 359us) server_negotiation.cc:407] Connection header received
0621 02:43:54.635581 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:54.635586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:54.635660 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:54.635773 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:54.636501 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60518: BlockingRecv error: recv got EOF from 127.0.0.1:60518 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:43:54.852847 23026 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 58738a421931486aaa6878852f80c5f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:43:55.160632 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:55.154881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.0.97.61:37957)
0621 02:43:55.155449 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:55.155457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:55.155479 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:43:55.155724 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:55.155729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:55.155876 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:55.156011 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:55.156454 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.157131 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.158531 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.158943 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.159081 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:55.159545 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:55.160041 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:55.160198 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:55.160269 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 02:43:55.160347 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":301,"threads_started":1}
I20260621 02:43:55.171953 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:55.166647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.0.97.60:32999)
0621 02:43:55.167102 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:55.167107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:55.167143 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:43:55.167335 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:55.167339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:55.167477 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:55.167608 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:55.167895 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.168554 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.169936 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.170344 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.170443 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:55.170882 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:55.171348 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:55.171456 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:55.171568 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 02:43:55.171628 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":252,"threads_started":1}
I20260621 02:43:55.186306 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:55.177787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.0.97.62:37671)
0621 02:43:55.179955 (+ 2168us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:55.179963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:55.180520 (+ 557us) server_negotiation.cc:407] Connection header received
0621 02:43:55.180642 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:55.180649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:55.180845 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:55.181131 (+ 286us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:55.181589 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.182489 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.184149 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.184553 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.184691 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:55.185243 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:55.185663 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:55.185769 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:55.185919 (+ 150us) server_negotiation.cc:299] Negotiation successful
0621 02:43:55.186005 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2009,"thread_start_us":1886,"threads_started":1}
I20260621 02:43:55.187307 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:55.177779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.0.97.61:37957)
0621 02:43:55.178113 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:55.178123 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:55.180570 (+ 2447us) server_negotiation.cc:407] Connection header received
0621 02:43:55.180849 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:55.180856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:55.180950 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:55.181037 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:55.182157 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.183093 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.185252 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.185880 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.186003 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:55.186517 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:55.186639 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:55.186714 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:55.186917 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 02:43:55.186993 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260621 02:43:55.187556 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:55.177972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.0.97.60:32999)
0621 02:43:55.178135 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:55.178142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:55.180954 (+ 2812us) server_negotiation.cc:407] Connection header received
0621 02:43:55.181137 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:55.181143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:55.181234 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:55.181369 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:43:55.182045 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.182925 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.185219 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:55.185660 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:55.185798 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:55.186615 (+ 817us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:43:55.186747 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:43:55.186835 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:55.187211 (+ 376us) server_negotiation.cc:299] Negotiation successful
0621 02:43:55.187287 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:43:55.191548 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22748
I20260621 02:43:55.192519 22810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:55.299685 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 22748
I20260621 02:43:55.320178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22822
I20260621 02:43:55.321414 22884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:55.429662 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 22822
I20260621 02:43:55.449601 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 22896
I20260621 02:43:55.450721 22958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:43:55.556313 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 22896
2026-06-21T02:43:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4918 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:43:55 dist-test-slave-jkn7 krb5kdc[23112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:43:55 dist-test-slave-jkn7 krb5kdc[23112](info): set up 2 sockets
Jun 21 02:43:55 dist-test-slave-jkn7 krb5kdc[23112](info): commencing operation
krb5kdc: starting...
W20260621 02:43:56.673719 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/test-user.keytab.
Jun 21 02:43:56 dist-test-slave-jkn7 krb5kdc[23112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:43:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260621 02:43:56.920416 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43751
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43751,127.0.97.61:43735,127.0.97.60:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:43:57.242476 23125 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:43:57.242870 23125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:57.242996 23125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:57.243161 23125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:57.250042 23125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:57.250162 23125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:57.250258 23125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:57.250363 23125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:57.250452 23125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:57.259963 23125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43751,127.0.97.61:43735,127.0.97.60:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:57.261245 23125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:57.262574 23125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:57.274281 23130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:57.274968 23133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:57.274994 23131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:57.275470 23125 server_base.cc:1034] running on GCE node
I20260621 02:43:57.276221 23125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:57.277832 23125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:57.279062 23125 hybrid_clock.cc:648] HybridClock initialized: now 1782009837278989 us; error 45 us; skew 500 ppm
I20260621 02:43:57.279392 23125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:57 dist-test-slave-jkn7 krb5kdc[23112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:57.282722 23125 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260621 02:43:57.284241 23125 webserver.cc:460] Webserver started at http://127.0.97.62:38699/ using document root <none> and password file <none>
I20260621 02:43:57.284966 23125 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:57.285081 23125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:57.285478 23125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:57.288434 23125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/data/instance:
uuid: "812f2087ac3f484884cfb633cd4da2c2"
format_stamp: "Formatted at 2026-06-21 02:43:57 on dist-test-slave-jkn7"
server_key: "1d69259e5d3db037cb0d4e51cc0ae7bd"
server_key_iv: "91d6ec12cc7e4e474de99688f8a78391"
server_key_version: "encryptionkey@0"
I20260621 02:43:57.289234 23125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/wal/instance:
uuid: "812f2087ac3f484884cfb633cd4da2c2"
format_stamp: "Formatted at 2026-06-21 02:43:57 on dist-test-slave-jkn7"
server_key: "1d69259e5d3db037cb0d4e51cc0ae7bd"
server_key_iv: "91d6ec12cc7e4e474de99688f8a78391"
server_key_version: "encryptionkey@0"
I20260621 02:43:57.294373 23125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:43:57.297889 23140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:57.299223 23125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:43:57.299410 23125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/wal
uuid: "812f2087ac3f484884cfb633cd4da2c2"
format_stamp: "Formatted at 2026-06-21 02:43:57 on dist-test-slave-jkn7"
server_key: "1d69259e5d3db037cb0d4e51cc0ae7bd"
server_key_iv: "91d6ec12cc7e4e474de99688f8a78391"
server_key_version: "encryptionkey@0"
I20260621 02:43:57.299639 23125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:57.331233 23125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:57.335248 23125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:57.335532 23125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:57.354403 23125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43751
I20260621 02:43:57.354434 23191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43751 every 8 connection(s)
I20260621 02:43:57.355808 23125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/data/info.pb
I20260621 02:43:57.360705 23192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:57.365949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 23125
I20260621 02:43:57.366180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-0/wal/instance
I20260621 02:43:57.366627 388 external_mini_cluster.cc:1428] Setting key 37430fb477179a1de127647be620cd97
I20260621 02:43:57.367374 23192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:57 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260621 02:43:57.392428 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.370681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43751 (local address 127.0.0.1:46198)
0621 02:43:57.371905 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.371946 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:57.373815 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:57.374624 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:57.374632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:57.375343 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:57.375659 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:57.375676 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.378024 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.378029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:57.379132 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:57.379144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.380482 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.380488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:57.380636 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:57.381412 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:57.381434 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:57.383843 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:57.388407 (+ 4564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:57.388418 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:57.388428 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:57.388823 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:57.389349 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:57.389353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:57.389356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:57.389507 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:57.390561 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:57.390567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:57.390757 (+ 190us) client_negotiation.cc:769] Sending connection context
0621 02:43:57.390959 (+ 202us) client_negotiation.cc:240] Negotiation successful
0621 02:43:57.391375 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":1,"spinlock_wait_cycles":15616,"thread_start_us":868,"threads_started":1}
I20260621 02:43:57.392477 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.370678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.0.97.62:43751)
0621 02:43:57.371502 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:57.371512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:57.372070 (+ 558us) server_negotiation.cc:407] Connection header received
0621 02:43:57.374171 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:57.374176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:57.374414 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:57.374615 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:57.375867 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.377841 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.379291 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.380235 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.380423 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:57.384092 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:57.384105 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:57.384124 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:57.384167 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:57.388249 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:57.388965 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:57.388975 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:57.388980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:57.389184 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:57.389621 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:57.389627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:57.389631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:57.390404 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:57.390638 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:57.391169 (+ 531us) server_negotiation.cc:299] Negotiation successful
0621 02:43:57.391558 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":627,"thread_start_us":397,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260621 02:43:57.395496 23192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260621 02:43:57.396610 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.396232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48528)
0621 02:43:57.396394 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.396512 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:43:57.396921 23142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
I20260621 02:43:57.398510 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.398170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48544)
0621 02:43:57.398308 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.398393 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:43:57.398726 23192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43735: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260621 02:43:57.425034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43735
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43751,127.0.97.61:43735,127.0.97.60:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:43:57.448021 23192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } attempt: 1
I20260621 02:43:57.449110 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.448688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48546)
0621 02:43:57.448883 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.448988 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 02:43:57.450102 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.449748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48548)
0621 02:43:57.449886 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.449956 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:43:57.450311 23192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43735: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
I20260621 02:43:57.518594 23192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } attempt: 2
I20260621 02:43:57.519718 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.519276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48564)
0621 02:43:57.519477 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.519577 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 02:43:57.520700 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.520334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48566)
0621 02:43:57.520503 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.520565 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:43:57.520875 23192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43735: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
I20260621 02:43:57.612185 23192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } attempt: 3
I20260621 02:43:57.613505 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.613001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48578)
0621 02:43:57.613228 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.613369 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260621 02:43:57.614511 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.614160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48580)
0621 02:43:57.614310 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.614381 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:43:57.614684 23192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43735: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
W20260621 02:43:57.745645 23199 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:43:57.745980 23199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:57.746099 23199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:57.746245 23199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:57.753417 23199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:57.753499 23199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:57.753582 23199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:57.753674 23199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:57.753783 23199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:57.757942 23192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } attempt: 4
I20260621 02:43:57.759198 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.758579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48592)
0621 02:43:57.758931 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.759038 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260621 02:43:57.760253 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.759872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48594)
0621 02:43:57.760021 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.760113 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:43:57.760449 23192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43735: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43735: connect: Connection refused (error 111)
I20260621 02:43:57.764164 23199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43751,127.0.97.61:43735,127.0.97.60:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:57.765492 23199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:57.766769 23199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:57.778522 23204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:57.778577 23207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:57.778611 23205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:57.779484 23199 server_base.cc:1034] running on GCE node
I20260621 02:43:57.780200 23199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:57.781589 23199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:57.782827 23199 hybrid_clock.cc:648] HybridClock initialized: now 1782009837782766 us; error 37 us; skew 500 ppm
I20260621 02:43:57.783146 23199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:57 dist-test-slave-jkn7 krb5kdc[23112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:57.786161 23199 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260621 02:43:57.787636 23199 webserver.cc:460] Webserver started at http://127.0.97.61:35103/ using document root <none> and password file <none>
I20260621 02:43:57.788321 23199 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:57.788404 23199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:57.788749 23199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:57.791885 23199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/data/instance:
uuid: "44864915d60e430ba6b1db3ad2369caa"
format_stamp: "Formatted at 2026-06-21 02:43:57 on dist-test-slave-jkn7"
server_key: "ad524aece208be1ce5e77cac8471187a"
server_key_iv: "bc16c6b020c380c27905e41514278f7d"
server_key_version: "encryptionkey@0"
I20260621 02:43:57.792680 23199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/wal/instance:
uuid: "44864915d60e430ba6b1db3ad2369caa"
format_stamp: "Formatted at 2026-06-21 02:43:57 on dist-test-slave-jkn7"
server_key: "ad524aece208be1ce5e77cac8471187a"
server_key_iv: "bc16c6b020c380c27905e41514278f7d"
server_key_version: "encryptionkey@0"
I20260621 02:43:57.797963 23199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 02:43:57.801692 23214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:57.803228 23199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:43:57.803421 23199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/wal
uuid: "44864915d60e430ba6b1db3ad2369caa"
format_stamp: "Formatted at 2026-06-21 02:43:57 on dist-test-slave-jkn7"
server_key: "ad524aece208be1ce5e77cac8471187a"
server_key_iv: "bc16c6b020c380c27905e41514278f7d"
server_key_version: "encryptionkey@0"
I20260621 02:43:57.803647 23199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:57.816576 23199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:57.820355 23199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:57.820618 23199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:57.838667 23199 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43735
I20260621 02:43:57.838686 23265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43735 every 8 connection(s)
I20260621 02:43:57.839989 23199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/data/info.pb
I20260621 02:43:57.843531 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:57.846975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 23199
I20260621 02:43:57.847168 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-1/wal/instance
I20260621 02:43:57.847594 388 external_mini_cluster.cc:1428] Setting key 877860c6c8229436cfcd5686ae5b3250
I20260621 02:43:57.849977 23266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:57 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260621 02:43:57.868815 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.852405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.0.97.62:43751)
0621 02:43:57.852618 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:57.852624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:57.853801 (+ 1177us) server_negotiation.cc:407] Connection header received
0621 02:43:57.855574 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:57.855580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:57.855672 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:57.855787 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:57.857189 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.857923 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.859363 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.859762 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.859894 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:57.863183 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:57.863188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:57.863191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:57.863228 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:57.865889 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:57.866598 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:57.866607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:57.866610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:57.866734 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:57.867139 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:57.867145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:57.867148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:57.867429 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:57.867596 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:57.868284 (+ 688us) server_negotiation.cc:299] Negotiation successful
0621 02:43:57.868502 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:43:57.869807 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.852836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43751 (local address 127.0.0.1:46200)
0621 02:43:57.853616 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.853665 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:57.855312 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:57.856007 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:57.856015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:57.856509 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:57.857010 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:57.857025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.858081 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.858085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:57.859211 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:57.859222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.859980 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.859986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:57.860202 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:57.860857 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:57.860883 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:57.862971 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:57.866033 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:57.866042 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:57.866057 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:57.866477 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:57.866842 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:57.866847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:57.866851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:57.867027 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:57.867554 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:57.867563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:57.867890 (+ 327us) client_negotiation.cc:769] Sending connection context
0621 02:43:57.868187 (+ 297us) client_negotiation.cc:240] Negotiation successful
0621 02:43:57.868651 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":331,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260621 02:43:57.871510 23266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:57 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260621 02:43:57.888111 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.872332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48604)
0621 02:43:57.872483 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.872503 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:57.872619 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:57.874789 (+ 2170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:57.874792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:57.875124 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:57.875254 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:57.875260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.877194 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.877197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:57.878073 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:57.878084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.879261 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.879265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:57.879562 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:57.880118 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:57.880145 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:57.882540 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:57.885694 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:57.885699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:57.885703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:57.886053 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:57.886436 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:57.886441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:57.886444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:57.886566 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:57.887144 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:57.887149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:57.887287 (+ 138us) client_negotiation.cc:769] Sending connection context
0621 02:43:57.887427 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 02:43:57.887711 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":54}
I20260621 02:43:57.888196 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.872594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.0.97.61:43735)
0621 02:43:57.874298 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:57.874308 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:57.874351 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:43:57.874406 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:57.874416 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:57.874632 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:57.874777 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:57.875424 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.877042 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.878205 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:57.879112 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:57.879259 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:57.882757 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:57.882767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:57.882778 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:57.882804 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:57.885539 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:57.886188 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:57.886196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:57.886200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:57.886311 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:57.886679 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:57.886687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:57.886690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:57.887008 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:57.887218 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:57.887572 (+ 354us) server_negotiation.cc:299] Negotiation successful
0621 02:43:57.887820 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":1525,"thread_start_us":1390,"threads_started":1}
I20260621 02:43:57.890829 23266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:43:57.892057 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.891624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45252)
0621 02:43:57.891808 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.891912 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:43:57.892311 23215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
I20260621 02:43:57.893862 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.893543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45258)
0621 02:43:57.893682 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.893760 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:43:57.894058 23266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260621 02:43:57.900192 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
/tmp/dist-test-taskXxBWLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37331
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43751,127.0.97.61:43735,127.0.97.60:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:43:57.943369 23266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } attempt: 1
I20260621 02:43:57.944461 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.944035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45264)
0621 02:43:57.944229 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.944317 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 02:43:57.945463 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:57.945077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45278)
0621 02:43:57.945237 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:57.945360 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:43:57.945693 23266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
I20260621 02:43:58.013921 23266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } attempt: 2
I20260621 02:43:58.015089 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.014548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45280)
0621 02:43:58.014775 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.014898 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 02:43:58.016106 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.015739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45292)
0621 02:43:58.015891 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.015974 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:43:58.016319 23266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
I20260621 02:43:58.059712 23192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } attempt: 5
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260621 02:43:58.074788 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.060454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48606)
0621 02:43:58.060679 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.060709 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:58.060840 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:58.061198 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:58.061203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:58.061763 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:58.061995 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.062008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.062979 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.062987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.064489 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.064503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.065236 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.065245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.065519 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.066332 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:58.066361 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:58.068996 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:58.072614 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.072618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.072621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.072979 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.073406 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.073409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.073411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.073532 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.074109 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:58.074112 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:58.074238 (+ 126us) client_negotiation.cc:769] Sending connection context
0621 02:43:58.074351 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 02:43:58.074534 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 02:43:58.075163 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.060581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.0.97.61:43735)
0621 02:43:58.060752 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.060756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.060775 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:43:58.060981 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.060986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.061047 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.061132 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.062149 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.062827 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.064640 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.065075 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.065173 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.069202 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.069212 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.069215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.069245 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.072496 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.073092 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.073098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.073101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.073241 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.073634 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.073639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.073642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.073995 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.074136 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.074486 (+ 350us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.074756 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:43:58.076227 23192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:43:58.077385 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.076922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45294)
0621 02:43:58.077098 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.077184 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 02:43:58.078338 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.078020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45304)
0621 02:43:58.078180 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.078240 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:43:58.078534 23192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
I20260621 02:43:58.107528 23266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } attempt: 3
I20260621 02:43:58.108671 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.108133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45316)
0621 02:43:58.108386 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.108488 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260621 02:43:58.109952 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.109415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45322)
0621 02:43:58.109674 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.109782 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260621 02:43:58.110195 23266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
I20260621 02:43:58.129725 23192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } attempt: 1
I20260621 02:43:58.130694 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.130277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45326)
0621 02:43:58.130460 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.130543 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 02:43:58.131637 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.131345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45338)
0621 02:43:58.131491 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.131549 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:43:58.131835 23192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
I20260621 02:43:58.200049 23192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } attempt: 2
I20260621 02:43:58.201140 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.200677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45354)
0621 02:43:58.200909 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.200993 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 02:43:58.202112 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.201801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45366)
0621 02:43:58.201947 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.202008 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 02:43:58.202288 23192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
W20260621 02:43:58.220784 23273 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:43:58.221166 23273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:43:58.221306 23273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:43:58.221448 23273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:43:58.228404 23273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:43:58.228514 23273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:43:58.228622 23273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:43:58.228713 23273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:43:58.228799 23273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:43:58.238265 23273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43751,127.0.97.61:43735,127.0.97.60:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:43:58.239861 23273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:43:58.241068 23273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:43:58.251979 23281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:58.252075 23278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:43:58.252055 23279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:43:58.252779 23273 server_base.cc:1034] running on GCE node
I20260621 02:43:58.253371 23266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } attempt: 4
I20260621 02:43:58.253489 23273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:43:58.254436 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.253936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45372)
0621 02:43:58.254161 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.254253 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 02:43:58.254997 23273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:43:58.255465 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.255134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45382)
0621 02:43:58.255294 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.255371 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 02:43:58.255715 23266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
I20260621 02:43:58.256238 23273 hybrid_clock.cc:648] HybridClock initialized: now 1782009838256168 us; error 39 us; skew 500 ppm
I20260621 02:43:58.256543 23273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:43:58.259464 23273 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260621 02:43:58.260778 23273 webserver.cc:460] Webserver started at http://127.0.97.60:39483/ using document root <none> and password file <none>
I20260621 02:43:58.261458 23273 fs_manager.cc:359] Metadata directory not provided
I20260621 02:43:58.261564 23273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:43:58.261860 23273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:43:58.264514 23273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/data/instance:
uuid: "a18319475a894c10bb78d025b22cae45"
format_stamp: "Formatted at 2026-06-21 02:43:58 on dist-test-slave-jkn7"
server_key: "81bac995ed2ab66059a837206cc22794"
server_key_iv: "d24ffb5dbbf3ea72c5bb75f5e8047708"
server_key_version: "encryptionkey@0"
I20260621 02:43:58.265270 23273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/wal/instance:
uuid: "a18319475a894c10bb78d025b22cae45"
format_stamp: "Formatted at 2026-06-21 02:43:58 on dist-test-slave-jkn7"
server_key: "81bac995ed2ab66059a837206cc22794"
server_key_iv: "d24ffb5dbbf3ea72c5bb75f5e8047708"
server_key_version: "encryptionkey@0"
I20260621 02:43:58.269867 23273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 02:43:58.273046 23288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:43:58.274246 23273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 02:43:58.274410 23273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/wal
uuid: "a18319475a894c10bb78d025b22cae45"
format_stamp: "Formatted at 2026-06-21 02:43:58 on dist-test-slave-jkn7"
server_key: "81bac995ed2ab66059a837206cc22794"
server_key_iv: "d24ffb5dbbf3ea72c5bb75f5e8047708"
server_key_version: "encryptionkey@0"
I20260621 02:43:58.274600 23273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:43:58.298741 23273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:43:58.303470 23273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:43:58.303696 23273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:43:58.308482 23192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } attempt: 3
I20260621 02:43:58.309809 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.309148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45390)
0621 02:43:58.309461 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.309571 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 02:43:58.311144 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.310616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45400)
0621 02:43:58.310877 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.310978 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260621 02:43:58.311401 23192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37331: connect: Connection refused (error 111)
I20260621 02:43:58.323007 23273 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37331
I20260621 02:43:58.323046 23339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37331 every 8 connection(s)
I20260621 02:43:58.324277 23273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/data/info.pb
I20260621 02:43:58.327807 23340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:43:58.332922 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu as pid 23273
I20260621 02:43:58.333107 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXxBWLs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009483336832-388-0/minicluster-data/master-2/wal/instance
I20260621 02:43:58.333642 388 external_mini_cluster.cc:1428] Setting key ab90e3bfc7009c4a73821d0a46e80dbe
I20260621 02:43:58.334704 23340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260621 02:43:58.356038 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.335395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.0.97.60:37331)
0621 02:43:58.336127 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.336138 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.336175 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:43:58.338114 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.338135 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.338530 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.338780 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.340442 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.342052 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.343758 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.344628 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.344792 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.349068 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.349080 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.349096 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.349147 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.352458 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.353164 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.353169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.353171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.353373 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.353813 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.353818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.353821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.354249 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.354529 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.354878 (+ 349us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.355109 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"spinlock_wait_cycles":14080,"thread_start_us":347,"threads_started":1}
I20260621 02:43:58.356928 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.337420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.0.97.62:43751)
0621 02:43:58.337627 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.337634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.338419 (+ 785us) server_negotiation.cc:407] Connection header received
0621 02:43:58.338759 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.338764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.338841 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.338948 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.340550 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.341646 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.343758 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.344417 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.344559 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.348514 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.348518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.348521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.348550 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.351964 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.352725 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.352729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.352731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.352852 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.353335 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.353340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.353343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.353716 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.353886 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.356432 (+ 2546us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.356661 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:43:58.357357 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.337477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43751 (local address 127.0.0.1:46206)
0621 02:43:58.338251 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.338296 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:58.338513 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:58.339032 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:58.339039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:58.339872 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:58.340371 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.340397 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.341818 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.341822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.343566 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.343580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.344618 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.344623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.344796 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.345643 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:58.345679 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:58.348327 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:58.352083 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.352091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.352099 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.352603 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.352961 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.352969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.352975 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.353205 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.353851 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:58.353859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:58.354152 (+ 293us) client_negotiation.cc:769] Sending connection context
0621 02:43:58.354416 (+ 264us) client_negotiation.cc:240] Negotiation successful
0621 02:43:58.354860 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"spinlock_wait_cycles":16768,"thread_start_us":395,"threads_started":1}
I20260621 02:43:58.364600 23340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260621 02:43:58.373353 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.358861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.0.97.62:43751)
0621 02:43:58.359017 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.359022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.359040 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:43:58.359131 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.359136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.359197 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.359280 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.360300 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.361445 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.362554 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.363180 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.363307 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.367638 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.367643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.367646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.367681 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.370448 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.371186 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.371191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.371194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.371333 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.371777 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.371782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.371788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.372051 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.372251 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.372590 (+ 339us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.372794 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 02:43:58.378172 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.365358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43735 (local address 127.0.0.1:48610)
0621 02:43:58.365624 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.365642 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:58.365757 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:58.366068 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:58.366076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:58.366466 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:58.366594 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.366602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.367581 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.367590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.368465 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.368476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.369448 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.369452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.369590 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.370097 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:58.370114 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:58.372291 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:58.375721 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.375727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.375730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.376080 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.376538 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.376544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.376548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.376725 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.377219 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:58.377224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:58.377402 (+ 178us) client_negotiation.cc:769] Sending connection context
0621 02:43:58.377540 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 02:43:58.377816 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260621 02:43:58.378400 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.365337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.0.97.61:43735)
0621 02:43:58.365499 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.365504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.365719 (+ 215us) server_negotiation.cc:407] Connection header received
0621 02:43:58.365863 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.365867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.365924 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.365999 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.366744 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.367426 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.368604 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.369283 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.369433 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.372485 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.372492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.372496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.372532 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.375588 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.376234 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.376239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.376245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.376410 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.376841 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.376844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.376846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.377089 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.377261 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.377677 (+ 416us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.377945 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":54}
I20260621 02:43:58.379586 23340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260621 02:43:58.390771 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.375305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.0.97.61:43735)
0621 02:43:58.375936 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.375940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.375958 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:43:58.375992 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.375995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.376042 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.376121 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.377638 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.378756 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.380413 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.380849 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.380986 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.385490 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.385495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.385498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.385526 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.388185 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.388809 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.388816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.388819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.388948 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.389338 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.389343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.389346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.389601 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.389872 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.390141 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.390351 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":526,"thread_start_us":316,"threads_started":1}
I20260621 02:43:58.392524 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.380583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.0.97.60:37331)
0621 02:43:58.380733 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.380739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.380752 (+ 13us) server_negotiation.cc:407] Connection header received
0621 02:43:58.380796 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.380798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.380843 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.380942 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.381722 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.382822 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.383942 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.384408 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.384505 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.387397 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.387402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.387405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.387430 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.389864 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.390462 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.390469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.390472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.390592 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.391001 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.391006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.391009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.391350 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.391459 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.391915 (+ 456us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.392188 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":53}
I20260621 02:43:58.392345 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.380358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45426)
0621 02:43:58.380523 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.380541 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:58.380632 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:58.380955 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:58.380959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:58.381395 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:58.381573 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.381586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.382974 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.382978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.383797 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.383807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.384573 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.384580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.384776 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.385361 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:58.385382 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:58.387228 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:58.390006 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.390012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.390016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.390352 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.390691 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.390695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.390700 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.390856 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.391469 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:58.391474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:58.391622 (+ 148us) client_negotiation.cc:769] Sending connection context
0621 02:43:58.391785 (+ 163us) client_negotiation.cc:240] Negotiation successful
0621 02:43:58.392045 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 02:43:58.395574 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:43:58.397804 23340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45: Bootstrap starting.
I20260621 02:43:58.397838 23167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "812f2087ac3f484884cfb633cd4da2c2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46208
I20260621 02:43:58.400960 23340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:58.402029 23340 log.cc:824] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:58.404877 23340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45: No bootstrap required, opened a new log
I20260621 02:43:58.411638 23340 raft_consensus.cc:348] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } }
I20260621 02:43:58.411960 23340 raft_consensus.cc:729] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18319475a894c10bb78d025b22cae45, State: Initialized, Role: FOLLOWER
I20260621 02:43:58.412364 23340 consensus_queue.cc:260] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } }
I20260621 02:43:58.413239 23347 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.413631 23347 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:58.414155 23340 sys_catalog.cc:564] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:43:58.422747 23358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:43:58.422883 23358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:43:58.421223 23273 master_runner.cc:186] Error getting master registration for 127.0.97.62:43751: OK, instance_id { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" instance_seqno: 1782009837348781 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:43:58.432188 23273 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:43:58.488627 23192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } attempt: 4
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260621 02:43:58.504684 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.489407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45434)
0621 02:43:58.489606 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.489626 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:58.489807 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:58.490179 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:58.490183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:58.490587 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:58.490751 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.490760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.492158 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.492165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.493068 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.493079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.494089 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.494095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.494303 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.494950 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:58.494991 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:58.497871 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:58.501858 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.501864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.501868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.502356 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.502812 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.502817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.502820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.502977 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.503667 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:58.503674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:58.503833 (+ 159us) client_negotiation.cc:769] Sending connection context
0621 02:43:58.503980 (+ 147us) client_negotiation.cc:240] Negotiation successful
0621 02:43:58.504272 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 02:43:58.504714 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.489457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.0.97.60:37331)
0621 02:43:58.489616 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.489624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.489726 (+ 102us) server_negotiation.cc:407] Connection header received
0621 02:43:58.489956 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.489961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.490044 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.490141 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.490902 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.492016 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.493210 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.493963 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.494076 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.498064 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.498070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.498076 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.498112 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.501733 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.502471 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.502477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.502481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.502659 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.503068 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.503073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.503077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.503531 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.503669 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.504120 (+ 451us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.504351 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:43:58.510706 23192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2: Bootstrap starting.
I20260621 02:43:58.514595 23192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:58.515822 23192 log.cc:824] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2: Log is configured to *not* fsync() on all Append() calls
I20260621 02:43:58.519045 23192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2: No bootstrap required, opened a new log
I20260621 02:43:58.526341 23192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } }
I20260621 02:43:58.526718 23192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812f2087ac3f484884cfb633cd4da2c2, State: Initialized, Role: FOLLOWER
I20260621 02:43:58.527187 23192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } }
I20260621 02:43:58.528141 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.528554 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:58.529292 23192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:58.529383 23167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 02:43:58.529557 23167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:58.533668 23167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } }
I20260621 02:43:58.536324 23167 leader_election.cc:288] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 44864915d60e430ba6b1db3ad2369caa (127.0.97.61:43735), a18319475a894c10bb78d025b22cae45 (127.0.97.60:37331)
I20260621 02:43:58.537155 23315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "812f2087ac3f484884cfb633cd4da2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a18319475a894c10bb78d025b22cae45"
I20260621 02:43:58.537678 23315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:58.541702 23315 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 812f2087ac3f484884cfb633cd4da2c2 in term 1.
I20260621 02:43:58.542626 23141 leader_election.cc:302] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 812f2087ac3f484884cfb633cd4da2c2, a18319475a894c10bb78d025b22cae45; no voters:
I20260621 02:43:58.543083 23361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:43:58.551443 23361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 812f2087ac3f484884cfb633cd4da2c2, State: Running, Role: LEADER
I20260621 02:43:58.552007 23361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } }
I20260621 02:43:58.554911 23266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } attempt: 5
W20260621 02:43:58.555984 23294 tablet.cc:2369] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:43:58.557991 23372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:43:58.558143 23372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 21 02:43:58 dist-test-slave-jkn7 krb5kdc[23112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260621 02:43:58.543059 23125 master_runner.cc:186] Error getting master registration for 127.0.97.61:43735: OK, instance_id { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" instance_seqno: 1782009837833284 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:43:58.568193 23125 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:43:58.572322 23375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 812f2087ac3f484884cfb633cd4da2c2. Latest consensus state: current_term: 1 leader_uuid: "812f2087ac3f484884cfb633cd4da2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.573040 23375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:58.573180 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.555504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37331 (local address 127.0.0.1:45450)
0621 02:43:58.555696 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 02:43:58.555717 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:43:58.555829 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:43:58.557438 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:43:58.557444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:43:58.557969 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:43:58.558194 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.558208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.559687 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.559692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.561172 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:43:58.561195 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.562370 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.562376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:43:58.562559 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.563224 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:43:58.563248 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:43:58.565993 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:43:58.570068 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.570075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.570078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.570623 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.571169 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:43:58.571177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:43:58.571180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:43:58.571365 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:43:58.572033 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:43:58.572039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:43:58.572221 (+ 182us) client_negotiation.cc:769] Sending connection context
0621 02:43:58.572388 (+ 167us) client_negotiation.cc:240] Negotiation successful
0621 02:43:58.572782 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:43:58.574296 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.556899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45450 (local address 127.0.97.60:37331)
0621 02:43:58.557064 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.557070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.557088 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:43:58.557167 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.557172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.557274 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.557389 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.558374 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.559526 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.561408 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.562040 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.562156 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.566171 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.566186 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.566191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.566227 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.569937 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.570761 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.570767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.570771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.571038 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.571506 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.571512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.571517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.571881 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.573550 (+ 1669us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.573701 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.573932 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:43:58.579983 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa: Bootstrap starting.
I20260621 02:43:58.583897 23266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa: Neither blocks nor log segments found. Creating new log.
I20260621 02:43:58.585819 23266 log.cc:824] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa: Log is configured to *not* fsync() on all Append() calls
W20260621 02:43:58.588255 23146 tablet.cc:2369] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:43:58.599707 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa: No bootstrap required, opened a new log
I20260621 02:43:58.605782 23381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:43:58.610122 23381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:43:58.621129 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.579099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.0.97.61:43735)
0621 02:43:58.579318 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.579324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.590131 (+ 10807us) server_negotiation.cc:407] Connection header received
0621 02:43:58.594865 (+ 4734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.594870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.594946 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.595012 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.602433 (+ 7421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.603515 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.605665 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.606342 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.606428 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.612777 (+ 6349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.612785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.612790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.612834 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.616732 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.618131 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.618139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.618154 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.618292 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.618956 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.618962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.618967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.619378 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.619630 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.620540 (+ 910us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.620780 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:43:58.627067 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.570824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.0.97.60:37331)
0621 02:43:58.575253 (+ 4429us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.575258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.588608 (+ 13350us) server_negotiation.cc:407] Connection header received
0621 02:43:58.590249 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.590255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.590344 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.590414 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.598376 (+ 7962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.599446 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.606226 (+ 6780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.606900 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.606996 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.618007 (+ 11011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.618017 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.618024 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.618072 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.621882 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.623633 (+ 1751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.623638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.623651 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.623789 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.624466 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.624471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.624474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.625085 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.625544 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.626381 (+ 837us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.626628 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4324,"thread_start_us":7318,"threads_started":1}
I20260621 02:43:58.627257 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:43:58.578571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.0.97.62:43751)
0621 02:43:58.578734 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.578739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.589052 (+ 10313us) server_negotiation.cc:407] Connection header received
0621 02:43:58.590622 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.590627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.590704 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.590775 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.596946 (+ 6171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.598188 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.606617 (+ 8429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:43:58.607353 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:43:58.607474 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:43:58.613144 (+ 5670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:43:58.613150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:43:58.613155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:43:58.613199 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:43:58.617257 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.623606 (+ 6349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.623611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.623614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.623790 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:43:58.624463 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:43:58.624468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:43:58.624471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:43:58.624840 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:43:58.625091 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0621 02:43:58.626074 (+ 983us) server_negotiation.cc:299] Negotiation successful
0621 02:43:58.626645 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:43:58.631280 23266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } }
I20260621 02:43:58.631670 23266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44864915d60e430ba6b1db3ad2369caa, State: Initialized, Role: FOLLOWER
I20260621 02:43:58.632303 23266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } }
I20260621 02:43:58.635567 23387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.636161 23315 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [term 1 FOLLOWER]: Refusing update from remote peer 812f2087ac3f484884cfb633cd4da2c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:58.636305 23387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:58.636861 23266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:43:58.636942 23241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "812f2087ac3f484884cfb633cd4da2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "44864915d60e430ba6b1db3ad2369caa"
I20260621 02:43:58.637116 23375 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:58.637499 23241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:43:58.640873 23241 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 812f2087ac3f484884cfb633cd4da2c2 in term 1.
I20260621 02:43:58.642120 23240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [term 1 FOLLOWER]: Refusing update from remote peer 812f2087ac3f484884cfb633cd4da2c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:43:58.643872 23375 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:43:58.660262 23387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 812f2087ac3f484884cfb633cd4da2c2. Latest consensus state: current_term: 1 leader_uuid: "812f2087ac3f484884cfb633cd4da2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.660574 23387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:58.666349 23347 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 812f2087ac3f484884cfb633cd4da2c2. Latest consensus state: current_term: 1 leader_uuid: "812f2087ac3f484884cfb633cd4da2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.666633 23347 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:58.669976 23386 mvcc.cc:205] Tried to move back new op lower bound from 7299112299038449664 to 7299112298715529216. Current Snapshot: MvccSnapshot[applied={T|T < 7299112299038449664}]
I20260621 02:43:58.678205 23395 mvcc.cc:205] Tried to move back new op lower bound from 7299112299038449664 to 7299112298715529216. Current Snapshot: MvccSnapshot[applied={T|T < 7299112299038449664}]
I20260621 02:43:58.681972 23347 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "812f2087ac3f484884cfb633cd4da2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.682309 23347 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:58.686949 23375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "812f2087ac3f484884cfb633cd4da2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.687245 23375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:58.691838 23381 catalog_manager.cc:1347] Generated new cluster ID: a2f4dc0793b64e0d8128562277f5d8df
I20260621 02:43:58.691941 23381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260621 02:43:58.692178 23402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:43:58.692301 23402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:43:58.696094 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "812f2087ac3f484884cfb633cd4da2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.696396 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:43:58.704733 23394 mvcc.cc:205] Tried to move back new op lower bound from 7299112299038449664 to 7299112298715529216. Current Snapshot: MvccSnapshot[applied={T|T < 7299112299038449664}]
I20260621 02:43:58.719004 23387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "812f2087ac3f484884cfb633cd4da2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812f2087ac3f484884cfb633cd4da2c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43751 } } peers { permanent_uuid: "44864915d60e430ba6b1db3ad2369caa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43735 } } peers { permanent_uuid: "a18319475a894c10bb78d025b22cae45" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37331 } } }
I20260621 02:43:58.719266 23387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:43:58.765615 23381 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:43:58.767611 23381 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:43:58.791011 23381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 812f2087ac3f484884cfb633cd4da2c2: Generated new TSK 0
I20260621 02:43:58.795284 23381 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:43:58.936496 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:58.911112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.0.97.61:43735)
0621 02:43:58.911335 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.911345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.933460 (+ 22115us) server_negotiation.cc:407] Connection header received
0621 02:43:58.934820 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.934827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.934918 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.934996 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.936216 (+ 1220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48638: BlockingRecv error: recv got EOF from 127.0.0.1:48638 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260621 02:43:58.940682 23343 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:58.917629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.0.97.60:37331)
0621 02:43:58.917815 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.917821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.939012 (+ 21191us) server_negotiation.cc:407] Connection header received
0621 02:43:58.939250 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.939256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.939338 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.939408 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.940451 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45466: BlockingRecv error: recv got EOF from 127.0.0.1:45466 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 02:43:58.942565 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:58.931499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.0.97.62:43751)
0621 02:43:58.931762 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.931768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.938739 (+ 6971us) server_negotiation.cc:407] Connection header received
0621 02:43:58.941271 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.941296 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.941379 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.941457 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.942312 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46222: BlockingRecv error: recv got EOF from 127.0.0.1:46222 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260621 02:43:58.965003 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:58.962883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.0.97.61:43735)
0621 02:43:58.963085 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.963091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.963540 (+ 449us) server_negotiation.cc:407] Connection header received
0621 02:43:58.963754 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.963760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.963899 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.964015 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.964783 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48654: BlockingRecv error: recv got EOF from 127.0.0.1:48654 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 02:43:58.975327 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:58.971186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.0.97.61:43735)
0621 02:43:58.971383 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.971389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.973783 (+ 2394us) server_negotiation.cc:407] Connection header received
0621 02:43:58.973979 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.973985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.974076 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.974183 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.975062 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48656: BlockingRecv error: recv got EOF from 127.0.0.1:48656 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260621 02:43:58.982928 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:58.981200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.0.97.61:43735)
0621 02:43:58.981387 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.981393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.981517 (+ 124us) server_negotiation.cc:407] Connection header received
0621 02:43:58.981768 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.981774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.981855 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.981956 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.982688 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48664: BlockingRecv error: recv got EOF from 127.0.0.1:48664 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260621 02:43:58.992656 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:58.990694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.0.97.61:43735)
0621 02:43:58.990870 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:58.990876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:58.991361 (+ 485us) server_negotiation.cc:407] Connection header received
0621 02:43:58.991518 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:58.991525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:58.991607 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:58.991716 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:58.992436 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48672: BlockingRecv error: recv got EOF from 127.0.0.1:48672 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 02:43:59.005712 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:59.003729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.0.97.61:43735)
0621 02:43:59.003892 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:59.003898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:59.004363 (+ 465us) server_negotiation.cc:407] Connection header received
0621 02:43:59.004549 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:59.004559 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:59.004657 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:59.004767 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:59.005513 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48678: BlockingRecv error: recv got EOF from 127.0.0.1:48678 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260621 02:43:59.027750 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:59.025598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.0.97.61:43735)
0621 02:43:59.025805 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:59.025810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:59.026326 (+ 516us) server_negotiation.cc:407] Connection header received
0621 02:43:59.026509 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:59.026515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:59.026603 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:59.026713 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:59.027458 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48682: BlockingRecv error: recv got EOF from 127.0.0.1:48682 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260621 02:43:59.073386 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:59.070865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.0.97.61:43735)
0621 02:43:59.071057 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:59.071064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:59.071601 (+ 537us) server_negotiation.cc:407] Connection header received
0621 02:43:59.071983 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:59.071990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:59.072092 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:59.072220 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:59.073103 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48684: BlockingRecv error: recv got EOF from 127.0.0.1:48684 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260621 02:43:59.143059 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:59.140991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.0.97.61:43735)
0621 02:43:59.141224 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:59.141230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:59.141584 (+ 354us) server_negotiation.cc:407] Connection header received
0621 02:43:59.141794 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:59.141798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:59.141896 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:59.141986 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:59.142841 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48694: BlockingRecv error: recv got EOF from 127.0.0.1:48694 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260621 02:43:59.277347 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:59.275364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.0.97.61:43735)
0621 02:43:59.275621 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:59.275628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:59.275874 (+ 246us) server_negotiation.cc:407] Connection header received
0621 02:43:59.276110 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:59.276115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:59.276188 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:59.276300 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:59.277037 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48706: BlockingRecv error: recv got EOF from 127.0.0.1:48706 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 02:43:59.424839 23358 catalog_manager.cc:1259] Loaded cluster ID: a2f4dc0793b64e0d8128562277f5d8df
I20260621 02:43:59.424942 23358 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45: loading cluster ID for follower catalog manager: success
I20260621 02:43:59.427424 23358 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45: acquiring CA information for follower catalog manager: success
I20260621 02:43:59.429212 23358 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a18319475a894c10bb78d025b22cae45: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:43:59.540235 23345 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:43:59.538375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.0.97.61:43735)
0621 02:43:59.538583 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:43:59.538591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:43:59.538844 (+ 253us) server_negotiation.cc:407] Connection header received
0621 02:43:59.539127 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:43:59.539140 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:43:59.539228 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:43:59.539347 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:43:59.540043 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48722: BlockingRecv error: recv got EOF from 127.0.0.1:48722 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:43:59.694247 23402 catalog_manager.cc:1259] Loaded cluster ID: a2f4dc0793b64e0d8128562277f5d8df
I20260621 02:43:59.694352 23402 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa: loading cluster ID for follower catalog manager: success
I20260621 02:43:59.696729 23402 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa: acquiring CA information for follower catalog manager: success
I20260621 02:43:59.698437 23402 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44864915d60e430ba6b1db3ad2369caa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:44:00.064507 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:00.058848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.0.97.61:43735)
0621 02:44:00.059536 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:00.059542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:00.059566 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:44:00.059645 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:00.059651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:00.059824 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:00.059959 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:00.060395 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.061079 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.062410 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.062811 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.062941 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:00.063410 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:00.063921 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:00.064081 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:00.064138 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 02:44:00.064216 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":459,"threads_started":1}
I20260621 02:44:00.076179 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:00.070330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.0.97.60:37331)
0621 02:44:00.070774 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:00.070781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:00.070874 (+ 93us) server_negotiation.cc:407] Connection header received
0621 02:44:00.071122 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:00.071126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:00.071281 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:00.071453 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:00.071853 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.072527 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.073906 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.074305 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.074441 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:00.074935 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:00.075507 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:00.075697 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:00.075831 (+ 134us) server_negotiation.cc:299] Negotiation successful
0621 02:44:00.075916 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":232,"threads_started":1}
I20260621 02:44:00.089876 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:00.082123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.97.61:43735)
0621 02:44:00.082347 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:00.082356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:00.084019 (+ 1663us) server_negotiation.cc:407] Connection header received
0621 02:44:00.084298 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:00.084303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:00.084383 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:00.084469 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:00.085361 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.086411 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.088100 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.088586 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.088716 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:00.089087 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:00.089227 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:00.089338 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:00.089493 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 02:44:00.089573 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:44:00.090968 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:00.082062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.0.97.62:43751)
0621 02:44:00.082776 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:00.082786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:00.083839 (+ 1053us) server_negotiation.cc:407] Connection header received
0621 02:44:00.084280 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:00.084286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:00.084500 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:00.084685 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:00.085490 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.086189 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.088487 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.089045 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.089180 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:00.089749 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:00.090225 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:00.090342 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:00.090574 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 02:44:00.090676 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":732,"threads_started":1}
I20260621 02:44:00.091298 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:00.082572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.0.97.60:37331)
0621 02:44:00.082793 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:00.082801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:00.084290 (+ 1489us) server_negotiation.cc:407] Connection header received
0621 02:44:00.084462 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:00.084466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:00.084545 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:00.084640 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:00.085171 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.086329 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.089338 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:00.089914 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:00.090040 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:00.090606 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:00.090702 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:00.090799 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:00.090934 (+ 135us) server_negotiation.cc:299] Negotiation successful
0621 02:44:00.091012 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:44:00.096915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 23125
I20260621 02:44:00.097895 23187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:00.211561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 23125
W20260621 02:44:00.232262 23289 connection.cc:535] server connection from 127.0.0.1:45434 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 02:44:00.232903 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 23199
I20260621 02:44:00.234037 23261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:00.344487 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 23199
I20260621 02:44:00.369661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu pid 23273
I20260621 02:44:00.370852 23335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:00.481566 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXxBWLs/build/asan/bin/kudu with pid 23273
2026-06-21T02:44:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4921 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9840 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357102 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST